Itera Ha: User Guide

iTERA HA
Ve r s i o n 6 . 0
User Guide
September 12, 2011
iTERA HA Version 6.0 User Guide
Copyright Vision Solutions® , Inc. 2003–2011

All rights reserved.
The information in this document is subject to change without notice and is furnished under a license agreement. This document
is proprietary to Vision Solutions, Inc., and may be used only as authorized in our license agreement. No portion of this manual
may be copied or otherwise reproduced without the express written consent of Vision Solutions, Inc.
Vision Solutions provides no expressed or implied warranty with this manual.
The following are trademarks or registered trademarks of their respective organizations or companies:
• Vision Solutions is a registered trademark and ORION Solutions, Integrator, Director, Data Manager, Vision Suite,
ECS/400, OMS/400, ODS/400, SAM/400, Replicate1, EchoCluster, EchoStream, RecoverNow and iTERA HA are
trademarks of Vision Solutions, Inc.
• DB2, IBM, i5/OS, iSeries, System i, System i5, AIX5L, Informix, System p, System x, and System z, and WebSphere—
International Business Machines Corporation.
• Adobe and Acrobat Reader—Adobe Systems, Inc.
• Double-Take, GeoCluster, and NSI—NSI Software, Inc.
• HP-UX—Hewlett-Packard Company.
• Teradata—Teradata Corporation.
• Intel—Intel Corporation.
• Java, all Java-based trademarks, and Solaris—Sun Microsystems, Inc.
• Linux—Linus Torvalds.
• Microsoft and Windows—Microsoft Corporation.
• Mozilla and Firefox—Mozilla Foundation.
• Netscape—Netscape Communications Corporation.
• Oracle—Oracle Corporation.
• Red Hat—Red Hat, Inc.
• Sybase—Sybase, Inc.
• Symantec and NetBackup—Symantec Corporation.
• UNIX and UNIXWare—the Open Group.
All other brands and product names are trademarks or registered trademarks of their respective owners.
If you would like technical assistance, please contact the Vision Solutions CustomerCare team.
Contents
Purpose of this guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Changes in this guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Professional Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Chapter 1—Introduction to iTERA HA High Availability . . . . . . . . . . . 3

Overview of iTERA HA High Availability Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Journaling Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
iTERA HA User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
HA User Profile Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
How to change the user profile password in iTERA HA . . . . . . . . . . . . . . . . . . . . . . . . . . 8
iTERA HA Installed Libraries and Other Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Accessing iTERA HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
How iTERA HA Works – Remote Journaling Replication Diagram. . . . . . . . . . . . . . . . . . . . . . . . 12
iTERA HA PTFs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Chapter 2—Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

System Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Menu System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Monitoring and Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Environment Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Other Menu Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
iTERA HA Subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Starting the iTERA HA Subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Menu Options to Start Subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
iTERA HA Start and End Subsystem Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Checking the Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Subsystem Jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Primary Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Jobs That Should Always Be Running On The Primary . . . . . . . . . . . . . . . . . . . . . . . . . 21
iTERA HA v6.0 User Guide iii

Contents
Required Scheduled Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Additional Jobs that Could be Running on the Primary Based on Replication Options 22
Target Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Jobs That Should Always Be Running On The Target. . . . . . . . . . . . . . . . . . . . . . . . . . 23
Required Scheduled Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Additional Jobs that Could be Running on the Target Based on Replication Options . 26
Work with Monitored Jobs (2.21) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Chapter 3—Understanding And Maintaining The Journaling

Technology Used By iTERA HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
JRM Default Parameters (3.31) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Start And Set The Journal Manager Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
JRM Managed Journals (3.32) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Modify The Journal Manager Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
JRM Job Schedule Entry (3.33) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Local Journals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Work with Local Journals (3.1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Create A New Mirror Journal (3.1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Define a User Journal to iTERA HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Journal Receivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Work with Journal Receivers (3.2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Primary Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Target Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Remote Journal Receivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Work with Remote Journaling (Primary) (3.3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Monitor Journal Entries (3.5) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Journaled Object Change Requests (3.22) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Apply Process - Work with Apply Jobs (3.4 - Target). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Journal Apply Statistics (1.1 F9 or 3.6, Target) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Chapter 4—Replication Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Replication Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Libraries That Should Not Be Replicated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Essentials Needed for Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Non-Mirrored Objects (4.30). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Set Up Non-Mirrored Library Object Syncing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
User Profile Replication (5.1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Replicate User Profiles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Object Lock Analysis and Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
iv iTERA HA v6.0 User Guide

Contents
Lock Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Object Sync Filter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Omit Filter Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
How to Create an Object Filter Entry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Filter Objects From User Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Sync Defined Non-Mirrored Objects By Library (4.31 - SYNCNMLIB) . . . . . . . . . . . . . . . . . 62
Sync Non-Mirrored Objects (4.32 - SYNCOBJ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Import Information From SAT Lite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Library Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Work with Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Library Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
How determine whether to use the Large Library Syncing Procedures . . . . . . . . . . . . . . . . 66
Steps to perform prior to a sync or resync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Specify a new library for mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Network Sync a Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Tape Sync a Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Steps to perform after a sync or resync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Sync a Library That Needs Objects Changed to a Different Journal Than the One Assigned to
the Library. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Remove a Library From Being Mirrored by iTERA HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Change the Mirror Journal Assigned to the Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Change the Journal to Which a Library and All Its Objects is Being Journaled . . . . . . . . . . 73
Replicate a Library to an ASP Other Than ASP 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Object Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
How to Change the Max Network Sync Size Parameter. . . . . . . . . . . . . . . . . . . . . . . . . 76
Resync an Object (or Objects) Via the Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
How to Determine Whether to Use the Large Object Sync Procedure . . . . . . . . . . . . . . . . . 77
Large Object Resync Via Network or Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Change the Journal to Which an Object in a Library is Being Journaled . . . . . . . . . . . . . . . 83
Change the Journal Image Being Used by the Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
System Library AutoSync (4.10) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Chapter 5—Non-Library Replication . . . . . . . . . . . . . . . . . . . . . . . . . 93

Enabling Replication Options (30.23) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Option 8=Start Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Non-Library Replication Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
User Profile (5.1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Directory Entries Replication (5.7) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Automatic Replication of Directory Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
iTERA HA v6.0 User Guide v

Contents
Conversion Utility for Directory Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

IFS Replication (5.2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Important Information About Replicating IFS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Journaling or Object-level Replication – How to Choose . . . . . . . . . . . . . . . . . . . . . . . . . 104
Replication Mapping Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Option 19 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
DSPF ‘/’ Output:. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
What to Replicate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
IFS Object Types Ineligible for Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Replication Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Create an IFS Journal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Journaling Defaults, Replication Options, and Assigning a Default Journal . . . . . . . . 112
Replicate IFS Using Journaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Replicate IFS Using Object-Level Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
IFS Audit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Purge Audit Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Ending IFS Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Out Queue & Spool File Replication (5.3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Spool File Replication Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Configuration Replication (5.4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Replicating Configuration Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Creating Maps to Globally Replicate Groups of Configuration Components . . . . . . . . . . 124
Selecting Individual Configuration Components for Replication . . . . . . . . . . . . . . . . . . . . 126
Ending Replication for Configuration Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Job Scheduler Replication (5.5) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
How to Set Up Job Scheduler Replication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Other Job Scheduler Replication Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Manage Scheduled Target Jobs From the Primary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Create a New Job on the Primary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Job Schedule Replication Errors (5.5 Target). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Duplicate jobs in the Job Scheduler Replication Screen (5.5) . . . . . . . . . . . . . . . . . . . . . . 135
Ending Job Scheduler Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Role Swap Considerations for Replicated Jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Other Scheduled Items That Can Be Replicated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
WebSphere MQ Replication (5.6) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
WebSphere MQ Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
WebSphere MQ Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
WebSphere MQ Replication Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
vi iTERA HA v6.0 User Guide

Contents
Ending WebSphere MQ Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Role Swap Procedures for WebSphere MQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
System Values Replication (5.8) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Work with Output Queue – E2WRKOUTQ (5.22) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Chapter 6—Monitoring Your System . . . . . . . . . . . . . . . . . . . . . . . . 145

Monitoring Your Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
System Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
How to Change the System Monitor Refresh Time Interval. . . . . . . . . . . . . . . . . . . . . . . . . 148
Mirroring Process Monitor (1.1 F16) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Objects Requesting Sync (1.1 F6) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
iTERA HA Message Log (1.1 F11 or E2MSGLOG). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Message Routing and Message Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Role Swap Readiness Monitor (1.1 F14 or 1.7) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Types of Tests Performed - Option 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Chapter 7—Monitoring Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

Daily Tasks Section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Weekly Tasks Section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Monthly Tasks Section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Mirrored Object Check (Option 1 on the OBJSNCSTS test in the Role Swap Readiness Moni-
tor) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Chapter 8—Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Audit Command Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Option 2=Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Managing the Audits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Scheduled Audits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
How to Create a Job Schedule Entry for an Audit . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Audit Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
How to Run the V6R1 OS Upgrade Viability Audit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Other Audits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
User Profile Component Checker (1.50.2 – Both) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Job Description Component Checker (1.50.3 – Both) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Chapter 9—Role Swap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Four Classifications of Role Swap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Virtual Role Swap Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
iTERA HA v6.0 User Guide vii

Contents
Virtual Role Swap with Communications Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

Role Swap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Preparation and Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
How to Determine the Type of Role Swap to Perform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Role Swap Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Two-node CRG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Three-node CRG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Role Swap Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Section One: Role Swap Pre-Planning Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Section Two: Pre-Role Swap Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Section Three: Final Pre-Role Swap Checks (30 minutes before role swap) . . . . . . . . . . . 210
Section Four: User Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Section Five: Perform the Role Swap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Section Six: Post Role Swap Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Section Seven: User Signon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Section Eight: Final Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Virtual Role Swap Test and Virtual Role Swap with Communications Test . . . . . . . . . . . . . . . 222
Execute the Virtual Role Swap Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
End Virtual Role Swap; Resume Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
How to Promote Backup 2 to Backup 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
How to Check the Progress of the Virtual Role Swap Recovery . . . . . . . . . . . . . . . . . . . . 234
Chapter 10—How to Get and Apply iTERA PTFs. . . . . . . . . . . . . . . 235

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Product IDs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
How to Acquire iTERA PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Load and Apply PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Important Information: Permanently Applying PTFs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Other PTF Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Download PTFs to PC; Burn to Disc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Download the iTERA HA PTF CD Image From Vision Solutions Support Central Site. 245
How to Create a Virtual Optical Drive on the IBM i and Download the PTF ISO Image . . 246
How to Use FTP to Retrieve an iTERA HA PTF to your PC Then Transfer It to Your IBM i 248
Troubleshooting and Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
How to Remove or Correct an iTERA PTF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
How to Automate PTF Retrieval and Notification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
viii iTERA HA v6.0 User Guide

Contents
Appendix A—iTERA HA Scheduled Audits and Other Jobs . . . . . . 263

Other Scheduled Audits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
Appendix B—IBM OS Upgrade Guidelines, Instructions, and Compati-

bility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
iTERA HA OS Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Guidelines for upgrading IBM OS in relation to iTERA HA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Minimum IBM OS Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
IBM Cumulative Fix Package and PTF Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
IBM PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
iTERA PTFs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Running the Primary and Target nodes on different levels of the IBM OS . . . . . . . . . . . . . . . . 271
Steps to perform the OS Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Pre-OS Upgrade Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Post-Upgrade Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Appendix C—Application/Software Upgrades Affecting Mirrored Li-

braries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Procedure for Software Upgrades affecting Mirrored Libraries . . . . . . . . . . . . . . . . . . . . . . . . . 275
Appendix D—i5/OS Recommendations. . . . . . . . . . . . . . . . . . . . . . 279

System Name Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Change the *LOCAL RDB Entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
IPL Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Change DDM TCP/IP Attributes (CHGDDMTCPA). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
V5R4M5 and lower. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
V6R1M0 and later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
IBM PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Appendix E—Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

Role Swap Readiness Monitor Issue Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Resolve IP Connection Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
Routing Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
Websites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
Fixing User/Takeover IP Address Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
Troubleshooting the Apply Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
Valid reasons to reset the apply job. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
Avoidable situations that can result in resetting the apply job . . . . . . . . . . . . . . . . . . . . . . 310
Troubleshooting High Disk Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
iTERA HA v6.0 User Guide ix

Contents
Primary System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310

Target System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
Troubleshooting DDM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
Errors that indicate DDM Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
DDM Troubleshooting – Things to check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Other Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
Mirroring restarts after ending mirroring jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
Object Audit Results on Target Only Shows Objects Being Added, but None Deleted . . 320
x iTERA HA v6.0 User Guide

Overview
1
Purpose of this guide
This guide is designed to provide an overview of the essential elements for
understanding iTERA HA. It begins with basic terminology definitions, an
introduction and overview to iTERA HA, and then provides step-by-step
instructions essential for configuring the product and initiating replication. It
continues with instruction on implementing the required processes that are
critical in maintaining, monitoring, and auditing iTERA HA. Additional
chapters include instructions for various procedures, such as role swaps, virtual
role swaps, and obtaining and applying iTERA HA PTFs. Consult the iTERA
HA v6.0 Reference Guide for descriptions of screen options, functions and
processes available within iTERA HA.
This guide should be used both as a tool for initial product training and as a
resource for monitoring and maintenance of your iTERA HA environment.
Changes in this guide

The documentation for iTERA HA v6.0 has been realigned. This user guide
now incorporates most content from the former iTERA HA v6.0 Training
Guide and the former iTERA HA v6.0 User and Troubleshooting Guide. The
Monitoring Checklist, Role Swap Checklist, and Virtual Role Swap Checklists
have been incorporated, as well as instructions for getting and applying iTERA
HA PTFs. Procedures and functionality that are considered advanced were
moved to the iTERA HA v6.0 Advanced Features Guide.
Contact Information
For information about obtaining training on iTERA HA, or if you need help
with any of our products, please contact Vision Solutions CustomerCare at
iTERA HA v6.0 User Guide 1

800-337-8214 or 949-724-5465 request from our website at
www.visionsolutions.com or by email to support@visionsolutions.com.
If you would like additional information about any of Vision Solutions’ other
data management products, please contact our Customer Sales department.
Professional Services
The following items are considered consulting and are offered as a billable
service:
• Role Swap
• Virtual Role Swap Test
• System Health Check
• System Migrations (hardware replacement)
• IP address changes
• IP connection issues
• Full system resynchronization
• Adding new nodes or CRGs
• Removing nodes or CRGs
• Additional product education
Services are scheduled based on availability. Allow at least two weeks for your
request to be scheduled.
2 iTERA HA v6.0 User Guide

Introduction to iTERA HA High
Availability 1
Overview of iTERA HA High Availability Terminology

The following terms are basic to achieving an understanding of the iTERA
HA product.
Term Description
The process of duplicating an object or library

Synchronization
between the primary node and the target node.
The iTERA HA process of using journaling to

record changes to selected objects (files, data areas,
Mirroring data queues) on the primary node then updating
the corresponding objects on the target node so
that they are identical.
Similar to synchronization, but it does not include

the mirroring of objects to detect changes. This
process ensures that objects on the primary node
that must exist on the target are replicated but the
objects are not monitored for any changes. For
Non-Mirrored Sync example, one use of the Non-Mirrored Sync is for
replicating user profile message queues. Typically,
we are not concerned about replicating the
messages in the queue, but the queue itself must be
replicated in order for the user profile to work on
the target node.
Also referred to as the source, or production node,

Primary Node it is the system on which daily production activity
occurs.
The system where an identical copy of selected

objects from the primary node reside, and where
change activity on the primary node is recorded by
the iTERA HA application in order to keep all
Target Node objects synchronized.
Target is a generic term for either a backup or
replicate node. (A target node may also be referred
to as a secondary, or mirrored box.)

Chapter 1: Introduction to iTERA HA High Availability
Term Description
A target node that can be used in a role swap or

Backup Node
failover.
A target node that is similar to a backup node in

that it can be mirrored to, but it cannot assume the
Replicate Node
role of a primary during a role swap. See “System
Roles” on page 15 for more information.
A Cluster Resource Group is an association of

nodes, which defines the relationship between the
nodes used in the replication process. The CRG is
created at the time iTERA HA is configured. Also
referred to as Resource Group (RG). Where
Cluster Resource
Group (CRG) needed, more than one CRG may be defined on a
node. For example, bi-directional replication
requires two CRGs. If you have separate divisions
or entities within your organization, a CRG may be
defined for each in order to ensure all resources
critical for those entities are replicated.
A Journal which records all changes, adds, and

Local Journal deletes to objects, then writes a record of what
changed to a journal receiver.
A record of a change made to an object. Changes

Journal Entry are recorded by a local journal then written to a
journal receiver
A component of the OS/400 journaling process

Journal Receiver that holds the journal entries generated by the local
journal with which it is associated.
A journaling process that sends a copy of journal

entries that have been written to journal receivers
Remote Journal
to identical journal receivers located on a remote
system.
An iTERA HA job that runs on the target node

that monitors journal receivers (which are attached
to remote journals) for new journal entries, then
Apply Job extracts and writes the data changes recorded in
these journal entries to the appropriate objects on
the target node. A separate apply job will exist for
each journal used by iTERA HA.
An object on the target node that no longer

Out-of-sync condition matches the corresponding object on the primary
node.
The same process as synchronization, but

performed when iTERA HA detects that an object
Resync condition
on the target node no longer matches the same
object on the primary node.

Overview of iTERA HA High Availability Terminology
Term Description
The process of having a backup node take on the

role of the primary node when you need to
temporarily have your normal primary node
unavailable (such as for OS upgrades, other vendor
software upgrades, full system backup, etc.)
Because of the iTERA HA mirroring process, data
Role Swap (Rollover)
selected for replication to the backup node will be
as current as the primary node. Therefore, both
users and processes can continue to work normally
after the role swap completes. All objects critical to
running the system must be replicated. A Role Swap
is typically a planned event
An abrupt ending of the primary node, due to a

system or hardware failure, which requires normal
Failover
production to now run on the backup node. A
failover is typically an unplanned event.
A journal created outside of the iTERA HA

product (either by third party software or by the
User Journal
client using the journal) for purposes other than
iTERA HA mirroring.
A journal that was created through the iTERA HA

Mirror Journal
product for use in the replication process.
Anything that resides in Auxiliary Storage on the

Object IBM i. Examples include files, programs,
commands, data areas, jobds, etc.

Journaling Basics
The following diagram depicts the basic concepts of IBM journaling. For more
information on journaling, see
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/rzaki/rzaki.pdf.

iTERA HA User Profiles
iTERA HA User Profiles

When iTERA HA was installed and configured on your system, the user
profiles HAxxADMIN and E2CSTMGR were also created.
The iTERA HA admin (HAxxADMIN) user profile is used for normal

monitoring and maintenance of the iTERA HA system. It is required for
performing role swaps, virtual role swaps, and failovers.
To view the iTERA HA Admin profile defined on your system, do the

following:
1. Select menu option 30.21, Cluster and Node Maintenance.
2. Select F7=Resource Groups.
3. Select option 2=Change for the CRG. (If you have more than one
CRG defined, repeat this step for the other CRGs.)
4. The profile is highlighted in the following screen:
E2CSTMGR is used internally within iTERA HA for Cluster Management, it

is never used for user sign on.
The profiles were set up with all the proper authorities needed to run the
various jobs.
IMPORTANT
Do not change the defined attributes of these profiles or iTERA HA will
cease to work properly!
– My iTERA HA User Profile: HA__ADMIN
Another profile that was automatically created for you is HAxxSUP. HAxxSUP
is a support profile which will allow access to the product using a language
other than the one defined for the CRG in order to facilitate assistance by
CustomerCare or other software service providers. The default language used
by the support profile is English, but it can be changed to any available

language. The support profile is automatically disabled upon creation and can
be enabled only when needed. The password for the profile is defined as
*FRCCHG, which will require a password change the first time it is used. To
assign a language to a profile, refer to the instructions in the iTERA HA v6.0
Advanced Features guide.
HA User Profile Requirements

In order for iTERA HA to keep your data in sync and running in the most
efficient manner, there are some key requirements that must be adhered to.
IMPORTANT
Failure to comply with these requirements will cause iTERA HA to
work incorrectly!
1. Passwords – During the syncing process, iTERA HA will be executing a

number of FTP and DDM jobs. Many of these jobs require a password in
order to execute the process. This password is stored in an encrypted
physical file.
IMPORTANT
If you need to change the iTERA HA password, do so ONLY within
iTERA HA.
If the subsystems are running, you can use the following procedure on the
primary system to change the password. This procedure will make changes
on the primary and the target and in WRKUSRPRF.
How to change the user profile password in iTERA HA

1. Select the Setup and Manage Environments menu (30.21).
2. Select F7 to access Resource Groups.
3. Select option 2=Change Set on the CRG you are changing the password for.
4. Select F9 (Password) to change the password:
a. Enter the current password.
b. Enter the new password.
c. Verify the new password.
d. Press Enter.
2. Special Authorities – In order for the iTERA HA profile, which is a

*USER class profile, to be able to manage certain functions, it must belong
to the ITERAOWNER profile group, which is a QSECOFR-equivalent
profile. The ITERAOWNER profile must have the following special
authorities:

iTERA HA Installed Libraries and Other Information
– *ALLOBJ
– *AUDIT
– *IOSYSCFG
– *JOBCTL
– *SAVSYS
– *SECADM
– *SERVICE
– *SPLCTL
3. DO NOT limit the capabilities of the iTERA HA user profile(s). These

profiles need to be replicated to the target with all authorities intact.
4. The HAxxADMIN profile may be copied for other users to administer

iTERA HA.
5. The ITERAOWNER profile and all profiles that belong to the

ITERAOWNER group should be closely guarded because of the
authorities they possess.
6. As a safeguard, you may want to set the initial menu for the
ITERAOWNER profile to *SIGNOFF after your installation of iTERA
HA is complete.

A two-character CRG code (typically “A1” if you have only one CRG defined)
was assigned to your systems when the software was installed. The CRG code is
included in the name of the CRG library and in other libraries and job names
throughout the program. “xx” is used throughout this guide to represent the
CRG code.
Also at installation, a subsystem was created both on the primary and the target
machines. This subsystem is named E2xxSBS (where xx is the CRG code as
indicated above).
Library
Library Description Name
(New iTERA HA
Installations)
Holds all of the iTERA HA ITHA

iTERA HA Base Library
programs and files.
holds all the CRG node-specific ITHAxx

information, programs, and files
CRG Library
that have been customized for
your environment.

Library
Library Description Name
(New iTERA HA
Installations)
Contains objects used in syncing, ITHAxxWRK

including data queues, and save
files, which are used to sync
Work Library objects and libraries. Once
synchronization completes
normally, these objects are deleted
from this library.
iTERA HA Journal and Local journal and journal receiver ITHAxxJRN

Journal Receiver library
Libraries
yyyy is the library number; ITHAxxyyyy

iTERA HA Journal and generally, 0100-0199 will be
Remote Journal found on the original backup and
Receiver Libraries 0200-0299 will be found on the
original primary node.
Used for journal receiver ITXP43

iTERA HA Journal management for several Vision
Manager Library products. It also contains
commands and custom programs.
iTERA Alert is the message ITALERT

iTERA Alert monitoring product installed with
iTERA HA.
Used to store selected IASP ITHAxxDD

definitions. (Most
implementations do not use this
library. This library is not
Data domain library automatically created for new
implementations.)
iTERA HA does not support
switchable IASPs.
ENU indicates English language ITHAENU,

libraries. If another language is ITXP43ENU,
loaded in iTERA HA, equivalent ITALERTENU
Local Language libraries will be created. The last
Enablement Libraries three characters of the library
name indicate the name of the
language (follows IBM naming
convention)
IMPORTANT
These libraries should not be replicated.

NOTE
Systems that upgraded from an earlier version of the product may use
ITE2 instead of ITHA. This guide and all other documentation will
refer only to ITHA.
Accessing iTERA HA
If you want a user to be able to access iTERA HA without using the iTERA
HA profile, do the following:
1. Add the following libraries to your library list (in this order):
Library Name
ITHAxx
ITHA
ITXP43
2. Grant the profile *USE authority to the libraries listed above.
3. Execute the command E2MAIN or E2MAINX.
– E2MAIN - Requires signoff to exit.

– E2MAINX - Allows F3 to exit.

How iTERA HA Works – Remote Journaling Replication

Diagram
The above illustration shows two environments. On the left is the production
or primary environment containing the objects to be mirrored. On the right is
the target (also referred to as a backup node) environment where the mirror
copy of objects are kept synchronized and ready to be used in the event of a
role swap or failover; i.e., the backup node becomes the production node.
The replication process occurs in several steps. Journaling (i.e., mirroring) on

the production node records data changes made to objects. This is shown in
the above diagram as the local journal on the production node which journals
files 1, 2, and 3 and records the data changes in the attached local journal
receiver. Objects that are journaled by iTERA HA include files, data queues,
and data areas.
While the journaling process records the data changes to journal receivers on
the production node, the remote journaling process also sends a copy of the
data changes (journal entries) to the target node via the network using TCP/IP.
(This is done by the operating system, independently of iTERA HA.) The
journal receivers on the primary node contain the actual data changes to the
mirrored objects, so that if for some reason communication between the
systems is interrupted, the i5/OS knows exactly what data was sent to the
target (and what data was not) by automatically restarting remote journaling at

iTERA HA PTFs
the appropriate journal receiver entry when communication is restored. It is in

this way that remote journaling automatically audits the transmission of data
between nodes.
Remote journaling consists of two components: 1) the “send” component

which resides on the production node, sending journal entries across the
network to the target node (illustrated by ‘Remote Journal Send,’ above); and
2) the “receive” component of the remote journal on the target node, which
receives the journal entries that have been sent by the “send” component.
Once the journal entries have been received by the receive component of the
remote journal on the target node, they are written to a journal receiver on the
target node.
The iTERA HA apply job then applies the data changes received by the journal
receiver on the target node to the appropriate objects on the target node. As
this is done, the file is kept synchronized with the production node.
When transactions are applied to objects on the target node, local journals on
the node record the changes to the mirrored objects, writing the transactions to
separate journal receivers, which are regularly deleted by iTERA HA. This
additional journaling process on the backup node is done to ensure that
journaling is active on that system and is ready to go when a Role Swap (or
Failover) is executed. This journaling will be needed upon execution of the
Role Swap (or Failover) to send changes to the former production system (the
new backup node) so that the mirroring process continues to take place. In
addition, there are ZZ Audits monitoring the local journal receivers in order to
ensure data integrity.
Remote journals are also associated with these contingency journals. However,
they remain inactive until a role swap occurs, at which time they will
automatically become activated. This ensures the entire mirroring process can
be quickly re-established between the new production node (formerly the
backup node), and the new backup node (formerly the production node).
Finally, the Object Monitor job processes the IBM system audit journal
QAUDJRN looking for object level changes (changes, deletes, creates,
renames, etc.) in mirrored libraries. If any of these conditions exist, then those
changes are replicated to the target node by iTERA HA.
iTERA HA PTFs
All updates and enhancements for iTERA HA v6.0 are done with IBM-style
PTFs. To successfully update iTERA HA, you must acquire the PTFs from
Vision Solutions’ FTP site, then load and apply them. See “How to Get and
Apply iTERA PTFs” on page 235 for complete instructions on PTF retrieval,
including alternative methods of obtaining PTFs, etc.

Vision Solutions releases a PTF Service Pack about once a month. Customers
subscribed to product notifications on the Vision Solutions Support Central
site will be notified when a Service Pack is available via e-mail.

Getting Started
2
System Roles
Node Description
The system where object-change activity is

Primary node
monitored by the iTERA HA application.
The system(s) where an identical copy of selected

objects from the primary node resides, and where
change activity on the primary node is recorded by
Backup node
the iTERA HA application in order to keep these
objects mirrored. A backup can be used in a role
swap, virtual role swap, and failover.
A replicate node is an additional node defined to

the CRG but is not part of the role swap
configuration. Objects are synchronized to a
replicate node just as they are for a backup node but
Replicate node
in a role swap or failover, a replicate node cannot
assume the role of a primary or backup. In all other
respects, a replicate node functions just as a backup
node.
NOTE
The term “target” is used to denote either a backup or replicate
node.

Chapter 2: Getting Started
Menu System
The Main Menu screen has been organized into sub-categories based on
iTERA HA processes with similar functionality.
Monitoring and Maintenance
Menu Option Description
Contains all monitoring functions that must be

1 – Monitoring Menu
performed on both a daily and weekly basis.
Contains all processing functions such as starting and

2 – Job Processing
Menu ending of the subsystems as well as a number of other
jobs that are available.
Contains all programs that have to do with

3 – Journal
Maintenance Menu journaling, journal receivers, remote journals, and
apply jobs.

4 – Library Replication
Menu replication of libraries and the objects within those
libraries.

Non-library replication. This includes User Profile
Replication, IFS, Device Configuration Replication,
Spool File Replication, Job Schedule Entry
Replication, Directory Entry Replication, etc.
5 – Non-Library
Replication Menu NOTE
The individual options need to be enabled
in the Replication Options screen (30.23)
in order for them to appear on this menu.

Menu System
6 – Auditing Command Manages all audits that should be performed on a

Console daily and/or weekly basis.
iTERA Alert is a message queue monitoring

notification tool. This menu option is not visible
7 – iTERA Alert until enabled. iTERA Alert configuration instructions
are available in the iTERA HA v6.0 Advanced
Features Guide.
Contains options for performing several system

10 – Tools Menu analysis functions and for retrieving and applying
PTFs.
Environment Management
Used for setting up and managing the iTERA HA

30 – Environment and environment. This includes tools used to ensure that
Setup Menu communications are set up and running as needed for
the iTERA HA system to work correctly.
Used to facilitate pre-role swap checks and the role

40 – Role Swap
Processing Menu swap, virtual role swap, and failover functions of
iTERA HA.
50 – System Control Provides access to configuration components.
Other Menu Features

• Fast Path – The menu system has been designed so that you can quickly
access any iTERA HA program by simply typing the path from any
iTERA HA command line. For example, if you want to access the Work
with Journal Receivers program, the long way to get there is to select option
3 from the main menu, and then press Enter. This displays the Journal
Maintenance Menu where you select option 2 and then press Enter again,
which will display the Journal Receiver Maintenance program. The FAST
PATH method allows you to simply type 3.2 (then enter) on any command
line within the iTERA HA system and it will take you to that program.
• Primary and secondary menus within iTERA HA will display the menu
option used to access it.
• Time and Date Stamps – Indicates when the menu option was last
accessed.
• iTERA HA commands – Any iTERA HA commands that can be used to
perform the same function as a menu option will be listed on the menu on
the right side of the description of the menu option in all capital letters.

Available menu options are based on the role the machine is assigned. When
iTERA HA was set up and configured in your environment, different roles
were assigned to the machines based on whether the machine was a primary
node or a target node (also referred to as production or source and backup).
IMPORTANT
Menu options differ based on the role assigned to the system.
iTERA HA Subsystems
You will have one subsystem per CRG on both the primary machine and the
target machine. If you have multiple CRGs defined, then you will have more
than one subsystem. The subsystem names are based on the two-character
CRG code designated during installation (e.g., E2A1SBS). All iTERA HA jobs
will run in the subsystem (unless otherwise noted during training).
IMPORTANT
When starting the subsystems, always start the target machine first,
then the primary machine. When ending the subsystems, always end
the primary machine first, then the target. Although iTERA HA will
be able to recover normal subsystem operation if this sequence is not
followed, it is recommended that you follow this procedure for
optimal streamlined performance.
Starting the iTERA HA Subsystems

Menu Options to Start Subsystems
From the iTERA HA Main Menu select option 2=Job Processing Menu.
From the Processing Menu select option 12=Start Mirroring Subsystem.
Fast Path
• Type 2.12 from any command line within the iTERA HA menu system
and it will start the subsystems for that machine.

iTERA HA Subsystems
iTERA HA Start and End Subsystem Commands

The command E2STRSBS can also be used to start up the subsystems and all
jobs that normally run in the subsystem.
IMPORTANT
This will need to be added to the startup procedure that runs after an
IPL (Initial Program Load).
Example of CL SBMJOB command:
SBMJOB CMD(ITHA/E2STRSBS) JOB(STR_ITERA)

JOBD(ITHAxx/E2JOBD) JOBQ(*LIBL/QCTL)
USER(HAA1ADMIN) INLLIBL(*JOBD)
Use the appropriate administrative profile for the CRG. For example,
if the CRG code is B1, then the admin profile for that CRG would
be HAB1ADMIN.
The command E2ENDSBS is used to end the iTERA HA subsystems and all
jobs running under that subsystem.
Fast Path
Type 2.13 from any command line within the iTERA HA menu system and it
will end the subsystems for that machine.
In order for the commands to work, the following libraries must be in your
library list (in this order): ITHAxx, ITHA, ITXP43.
IMPORTANT
You must start the subsystems on all nodes. Issuing commands or
menu options from just one node will NOT bring up all nodes.
Checking the Subsystem

Always check to make sure that the subsystems have started up normally and
that all jobs that should be running in the subsystems appear as well (nothing
should appear in a MSGW status, except if running iTERA Alert). There are a
number of ways that you can check the subsystems:
Menu Options to Check Subsystems

• From the iTERA HA Main Menu select option 2 to go to the Processing
Menu.
• From the Processing Menu, select option 11, Work with Subsystem Jobs.

Fast Path
• Type 2.11 from any command line within the iTERA HA menu system
and it will display the subsystems and all iTERA HA jobs running for that
machine.
iTERA HA View Subsystem Command

• E2SBS – Used to view the iTERA HA subsystem and all jobs running
within the subsystem.
NOTE
Different jobs will be running in the subsystems based on the role
that machine is actively performing.
The first two characters for most of the iTERA HA jobs is the two-character
CRG code (in this case, A1; most basic iTERA HA installations use A1).
Look for any jobs that are:
• In a MSGW status.
• Any jobs on hold.
• Any jobs that should be running but are not.
Subsystem Jobs
In the ITHAxx library on both nodes is a file called E2PJOBS. This file holds a
list of jobs that should be running in your subsystems based on the role that
machine is performing. When the subsystems start up, it will automatically
start those jobs.

Subsystem Jobs
NOTE
The following is not a comprehensive list of all jobs. Additional job
information is available in an appendix of the iTERA HA Reference
Guide.
Primary Jobs
The following jobs should always be running on the primary machine.
Additional jobs may be running in your system based on whether certain
features, such as Spool file replication, Configuration Replication, etc., have
been enabled.
Jobs That Should Always Be Running On The Primary
Job Name Description
Job for the Autonomic Heal technology. The job

processes the Heal requests. For more information,
xx_HEALNnn
see Autonomic Heal section of the iTERA HA v6.0
Advanced Features guide.
Java Send Remote Command. Runs a multi-threaded

xx_JSNDRC Java application that sends, receives, and executes
remote commands.
Object Monitor Job. This job uses the QAUDJRN

system journal to identify object-level changes in
xx_OBJMON1 mirrored libraries. These changes include creates,
deletes, changes, renames, moves from one library to
another, etc.
This job processes ZC entries, audit level changes,

authority audits, object authority changes, alternate
name audit, user profile changes, system value audit
and changes, save files. Processes audits that go
through the transport journal, such as source member,
xx_OBJMON2
library, and logical file attributes. Also submits a
purge job that removes old records from several files.
The purge is submitted at midnight or whenever the
job is restarted. The files purged are listed in the
50.11 screen (HISTORY and LOG).
xx_OBJMON3 Processes the same jobs as xx_OBJMON2.
This job initiates the individual tests contained in the

xx_RSRMON
Role Swap Readiness Monitor.
The remote journal transport syncing sending job(s)

xx_RJTSAA are used to replicate both mirrored and non-mirrored
objects to the target using remote journaling.

The Remote Command Monitor job monitors for

xx_RMTCMD any remote commands coming from the target system
that are to be run on the primary.
This job uses DDM to processes sync requests from

the E2POSR file which resides on the target machine
xx_SNC_Nnn
(nn is the two digit number of the target node
machine).
This job submits the MONSTS job, which updates

the information displayed in the System Monitor
screen, such as local journal status, remote journaling
xx_SYSMON
status, apply job status, and unapplied journal entries.
It is initiated every fifteen minutes but can be
adjusted, if needed (2.21, F8).
JSNDRCJAVA Java application started by the xx_JSNDRC job.
Required Scheduled Job
The Journal Manager job manages and cleans up

XPJRNMGT receivers. It runs every thirty minutes but the default
can be changed, if needed.
Additional Jobs that Could be Running on the Primary

Based on Replication Options
xx_DEVREP Device Configuration Replication
xx_DIRREP Directory Entry Replication
xx_IFSMON IFS Replication
xx_JBSREP Job Scheduler Replication

Subsystem Jobs
xx_RPTREP Spool File Replication
Permanent extra sync jobs are available to assist the

standard sync job with sync request processing. To
start permanent extra sync jobs, change the data area
E2OSRJOBS in the 10.50 Parameters screen to
indicate the desired number of additional jobs, then
end and restart the subsystems on all nodes.
When the standard sync job (named xx_SNC_Nnn)
starts in the iTERA HA subsystem, depending on the
value in the E2OSRJOBS data area, it submits one or
more additional jobs named xx_SN3_Nnn, where nn
is the sync job number. A value of 1 will cause one
additional sync job to be submitted, a value of 2 will
xx_SN3_Nnn cause two additional sync jobs to be submitted, etc.
The xx_SN3_Nnn sync jobs are persistent. They
continue to run and search for sync requests to
process every 15 seconds, just like the standard sync
job. They will end when the iTERA HA subsystem is
ended and only start the number of jobs indicated in
E2OSRJOBS when the subsystem is restarted.
Additional temporary sync jobs can be submitted by
pressing F16 from the 1.1 F6 screen, but they are only
active for as long as they are processing entries. As
soon as there are no more entries to process, the jobs
are ended.)
Target Jobs
The following jobs should be running continually on the target machines.
Additional jobs may be running in your subsystem based on whether certain
features, such as Spool file Replication and IFS, have been enabled.
Jobs That Should Always Be Running On The Target
Object Monitor Job. This job uses the QAUDJRN

system journal to keep track of restoration of libraries,
xx_OBJMON1
objects, etc., in order for the system to know when to
carry on and perform other functions.
The Role Swap Readiness Monitor job initiates the

xx_RSRMON
individual tests contained in the monitor.
Remote Command Monitor. This job monitors for

xx_RMTCMD any remote commands coming from the primary
system that are to run on the target.

Java Send Remote Command. Runs a multi-threaded

xx_JSNDRC Java application that sends, receives, and executes
remote commands.
Monitors the operating system for completion of

xx_APRBMON unique key access path rebuild, then notifies the apply
job that it can process the journal entries for files.
JSNDRCJAVA Java application started by xx_JSNDRC.
Apply job(s) for mirrored journals. Mirrored journals

are journals that were created for the iTERA HA
ZMxxJRNA
product. There will be an apply job for each mirrored
journal.
Apply job for transport journals. There will be one

ZMxxTJRA
transport journal apply job per transport journal.
The apply jobs for RJT syncing journals process the

journal entries from the xx_RJTSAA job on the
ZRxxSJNA
primary. There will be one ZR job for each RJT Sync
journal.

Subsystem Jobs
Apply jobs for user journals. This job only runs if user
journals (journals that have been set up by third-party
ZU_xxxx
software vendors or created outside of iTERA HA)
have been incorporated into iTERA HA.
The ZZ-Audit jobs audit the local journal receiver for

any changes that occur on the target that did not
originate from the primary. The ZZ-Audit functions
like a safety net. There have been a significant number
of instances where these jobs have identified data on
the target machine that had been modified by some
unauthorized process.
The ZZ-Audits are a major component of the Virtual
Role Swap, which allows you to perform a complete
test of the target environment’s processes that are
redundant (e.g., applications that are being
replicated).
The ZZ-Audit jobs run automatically and
ZZxxJRNA continuously on your system. There is one ZZ job for
each data journal apply process. They look for activity
on the target node that did not come from the apply
job. Data changed on the target node by any process
other than the iTERA HA apply job will cause the
object to be out of sync. If Autonomic Heal is active
for the object that has records in error, the errors will
be corrected. If not, a resync will be initiated.
Information about and instructions for enabling
Autonomic Heal are available in the iTERA HA 6.0
Advanced Features Guide.
The journals must have the FIXLENDTA parameter
set to *JOB *USR and *PGM in order for the ZZ
Audits to function properly.
Required Scheduled Jobs
Journal Manager Job. This job runs every thirty

minutes but this default can be changed, if
XPJRNMGT
needed. This job manages and cleans up journal
receivers.

Additional Jobs that Could be Running on the Target

Based on Replication Options
xx_JBSREP Job Scheduler Replication
xx_RPTCHG (V5R3 only) Spool File Replication
Spool File Replication (This job is only required on

xx_RPTREP
the target if system is an LPR.)
Executes the instructions for changes to authority and

xx_IFSCMD owner attributes for a directory being synced. (This
job may also run under the name DV_IFSCMD.)
Work with Monitored Jobs (2.21)

This screen displays many of the jobs that could potentially be started. Access
it by entering 2.21 from any iTERA HA command line.
IMPORTANT
You must end and restart the iTERA HA subsystems after making
changes on this screen.
Both primary and target jobs are displayed. The System column displays the
node to which the job applies.

Subsystem Jobs
The F8=Set Delay key displays the Job Monitor Control screen. Here you can
adjust how often the jobs for the Job Monitor, System Monitor, and Auto Sync
run.


Understanding And Maintaining
The Journaling Technology 3
Used By iTERA HA
This chapter discusses the journaling technology used in iTERA HA, which
includes local journals, journal receivers, remote journals, and apply process.
JRM Default Parameters (3.31)

This procedure must be done on all nodes.
• Fast path: Enter 3.31 on any iTERA HA command line.

• From the Main Menu, select option 3 – Journal Maintenance Menu.
Select option 31 – JRM Default Parameters.
• From the Journal Manager System Controls screen, you can configure
several values which act as defaults in other journal receiver management
screens. Since these are default settings, keep in mind that any journal
you create can have separate parameter settings.
NOTE
Doing this step on all systems prior to creating the first journal will
save steps.
Start And Set The Journal Manager Defaults

1. Select the JRM Default Parameters option (3.31).
2. Review the screen and the parameter descriptions in the table below and
set the values as indicated.

Chapter 3: Understanding And Maintaining The Journaling Technology Used By iTERA HA
The fields described below are the only fields on this screen that apply to
iTERA HA.
Field Description
Journal Change Indicates (in minutes) how often journal receivers will be
Frequency changed.
If you are just using journaling for mirroring functions

there is no need to require a save. However, some do like to
save receivers for auditing purposes. If Delete w/o save is
set to Y, the Receiver will be deleted after the number of
hours indicated in the Local Receiver Retention setting is
Local Receiver
Retention reached.
and An “N” in the Delete w/o Save field will not allow receivers
Delete w/o save
to be deleted until they are saved (even if they are older
than the hours specified in the Local Receiver Retention
field). Requiring a save will enable you to know what
changes have been made by comparing the receivers.
However, the save doesn’t automatically do the comparison
for you, it must be done manually.
Specifies the frequency of the run interval for journal

manager job. When this job runs, any journals that need to
be changed (based on the setting in the Journal Change
Frequency field) are changed. If Vision RecoverNow
(formerly iTERA Vault) is being used, all receivers may
Job Scheduler then be saved, including the attached receivers, and
Frequency transmitted to a remote storage platform. Any receivers that
have met conditions to be deleted are removed from the
System i (defined in the Local Receiver Retention field).
Pressing enter after making any changes on this screen will
replace the current job in the Job Scheduler and create new
one.
3. Press Enter. A job named XPJRNMGT will be created in the job

scheduler. After it runs, it reschedules itself to run again at the interval

JRM Managed Journals (3.32)
specified in the Job Scheduler Frequency field, unless it is either deleted or

held.
4. Select Job Scheduler (3.33) to review the job schedule entry.
JRM Managed Journals (3.32)

• FAST PATH: Enter 3.32 on any iTERA HA command line.

Select option 32 – JRM Managed Journals.
This screen displays all journals being used to replicate objects in iTERA HA.
It is primarily used for changing the settings for existing journals.
The following screen is displayed when selecting either F9 (JRM Defaults) or

option 2=Edit Options.
If changes are made, press Enter prior to exiting the screen.
Modify The Journal Manager Definition

1. Select JRM Managed Journals (3.32).
2. Select Edit options (option 2), press enter.

3. Set the Control Journal Management to Y (Journal Manager will manage

the journal’s receivers) or N (You will manage the journal’s receivers).
4. Set Journal Change Frequency (min).
5. Set Journal Receiver Retention (hrs).

6. Set Delete Receivers without save to Y if the Journal Manger can delete the
receivers before they are saved or N if the receivers must be saved before
being deleted.
7. Press enter to update.
JRM Job Schedule Entry (3.33)

• FAST PATH: Enter 3.33 on any iTERA HA command line

Select option 33 – Work with JRM Job Schedule Entry.
This option accesses the i5/OS Work with Job Schedule Entries screen, which
displays the scheduled job that is created by the system to manage the deletion
of journal receivers.
Local Journals
Journal Types
Mirror journals are typically created within iTERA for the purpose of
mirroring objects. (Although some user journals may be converted to mirror
journals, see below). When a mirror journal is assigned to a library, all
journaling functions (ending and starting) will be controlled by iTERA HA
processes. For example, adding a filter for an object will end journaling on the
object if it is being journaled by a mirror journal. Only mirror journals are
eligible to be assigned as a default journal.
User journals are journals that have been created outside of iTERA HA, either
for in-house use or for use by third-party applications. The objects are being

Local Journals
journaled for a purpose other than HA. User journals may be imported (or
defined) in iTERA HA in order to mirror the objects. Most journal
maintenance functions for user journals are restricted in iTERA HA.
Journaling is not automatically started or ended for objects in user journals. A
user-created user journal may be converted to a mirror journal so that it may be
fully-managed by iTERA HA. User journals are ineligible to be assigned as a
default journal. User journals created by third-party applications should not be
converted to mirror journals. Doing so may cause the application to not
function correctly
Work with Local Journals (3.1)

• FAST PATH: Enter 3.1 on any iTERA HA command line.
• From the Main Menu select option 3 – Journal Maintenance Menu.
Select option 1 – Work with Local Journals
• 1.1, F16, F14 – Local Journals can be accessed via the System Monitor.
This screen allows you to create and manage the local journals that are used by
iTERA HA for the mirroring process.
When accessed on the primary node, you are viewing the local journals that
journal the objects selected for mirroring.
When accessed on the target node, you are viewing the local journals on that
system. These local journals are active, but are not used in replication unless a
role swap or failover is executed. They are also read by the ZZ Audits, which
check for unauthorized updates performed on the target node.
Any changes to journals that affect both the primary and target nodes should
always be executed from 3.1 on the primary node.
Typically, the only time you need to view this menu from the backup node is
to ensure that everything is properly defined prior to a role swap.

5=Work Journal Attributes – Calls IBM’s WRKJRNA command:
Select F16 to access Work with Remote Journal Information.

Local Journals
Create A New Mirror Journal (3.1)

This procedure is executed on the primary node and will create the journal on
both the primary and the target nodes.
IMPORTANT
Verify that the subsystems are active on all nodes (E2SBS). Use 2.11
or E2STRSBS to start them.
NOTE
If you have not already set the JRM default parameters in 3.31, do so
before creating the first journal. This must be done on both the
primary machine and the target machine. See “JRM Default
Parameters (3.31)” on page 29 for more information.
1. Select 3.1 - Work with Local Journals on the primary node.
2. Select F6=Create Mirror Journal. Set the New Journal Type (use option
D=Data for the journals that will be used for library replication), then
press Enter.

3. Type in the description, then press Enter.
4. On the target machine, go to Apply Job Maintenance (3.4). Within fifteen

minutes, the job status should change to either a RUN status or an EVTW
status. However, by manually updating the System Monitor (1.1, primary,
F10=Update Monitor), you can expedite processing.

Journal Receivers
Define a User Journal to iTERA HA

NOTE
In order for the ZZ-Audits to function correctly for user journals, the
FIXLENDTA parameter for the journal must be set to
*JOBUSRPGM.
1. Select Work with Local Journals (3.1, primary).
2. Select F8=Load New User Journal.
3. Enter the library that contains the user journal, the user journal name and
a description, then press Enter.
4. Press F8=Confirm and Load.
5. If the Mirror Status is not ON, select 32=Toggle Mirror Status, then press
Enter.
6. Select Work with Apply Jobs (3.4, target). Ensure the Job Sts field for the
apply job does not display dashes. If it does, review the steps for
troubleshooting the apply job. Contact CustomerCare if unable to resolve.
IMPORTANT
QAUDJRN is not supported for data replication. It should never be
added to the Local Journal Maintenance screen, nor should it ever
have an attached remote journal or apply job. (QAUDJRN is read
and processed through the OBJMON jobs. Adding it to the Local
Journal Maintenance screen will cause conflicts with these jobs.)
Journal Receivers

Work with Journal Receivers (3.2)

• From the Main Menu select option 3 – Journal Maintenance Menu.

Select option 2 – Work with Journal Receivers.
• 1.1, F16, F7 – Journal Receivers can be accessed via the System Monitor.
This screen displays information about all journal receivers identified to

iTERA HA on each system. As transactions occur on the primary node, local
journals on the primary node record entries in their attached journal receivers.
Simultaneously, the entries are sent across the network through the remote
journal process to the target node.
• When you access the journal receivers option from the primary node, you
will initially view the status of the journal receivers that are attached to the
local journals on that system.
• When you access this option from the target node, you will initially view
the status of the journal receivers that are associated with the remote
journals.
Primary Node
F7 allows you to toggle between the Local and Remote Receivers on the
primary node.

Journal Receivers
NOTE
There are no attached receivers because a role swap has never been
executed in this environment.
Target Node
IMPORTANT
Always be aware whether you are viewing Local Receivers or Remote
Receivers.
The F7 key allows you to toggle from the view displaying the receivers attached
to the remote journals on this node to the receivers attached to the local
journals on this node.

Remote Journal Receivers
Work with Remote Journaling (Primary) (3.3)

Select option 3 – Work with Remote Journaling.
• 1.1, F16, F8.
This screen allows you to view and manage the remote journal components
that reside on the system you are on.
• When you initially access this option from the primary node, the status of
the “send” component of the remote journals is displayed.
• When you initially access this option from the target node, the status of
the “receive” component of the remote journals is displayed.
The primary node screens.

NOTE
Pay close attention to the direction of the remote journals and to
whether you are looking at the Send or the Receive portion of the
remote journal.
Select F7 to toggle between the remote journals replicating in the

opposite direction
Remote Journal Maintenance screen (3.3) on the target node.

Remote Journal Maintenance, target node, F7=Toggle Local Remote
Monitor Journal Entries (3.5)

This screen is primarily used to monitor the journal entries flowing through a
journal.
The F7 key is used to reset the starting point which includes retrieving the
current receivers being used by the journal, as well as the starting sequence
number. By resetting the start point, you will be able to determine how many
entries have been handled by each receiver.
Journaled Object Change Requests (3.22)

This option can be used to specify different journals for a single object or
groups of objects. It can also be used to specify a different type of journal entry
recording method (i.e. “after” journal entries, or “before-and-after” journal
entries). This is one of the best places in iTERA HA to do load balancing for
journals. This procedure is only applicable after replication has been initiated
and if objects need to be moved from one journal to another.
This procedure is commonly used when there are multiple objects to be moved
to a different journal. To transfer a single object to a different journal the Work

with Mirrored Objects screen (4.21, option 7=Transfer Journal) may be used
instead.
NOTE
An object cannot be moved from a mirror journal to a user journal or
from a user journal to a mirror journal.
1. Press the F18=Import Object Change Requests key to select the library
and/or objects to load into the Journal Change Requests screen.
2. On the Select Mirror Journals screen, select a journal using option

1=Select with Default Image.
3. Once the journal has been selected, the Journal Change Requests screen is
again displayed. A separate request is generated for each of the objects
within the library being changed. Select the appropriate option, to the
default, after images, or both before and after images, then press Enter.
The Journal Change Requests screen is displayed.

4. Press either option 9=Attempt Request for each object to be processed or

press F8=Submit Change Processor to submit all change requests.
5. Once the request is processed, the current and requested journal column
data will display yellow text. Press F7 to toggle between all, pending, and
completed change requests.
NOTE
Objects that are locked at the time the F8=Submit Change Processor
is executed will remain in a Pending status and iTERA HA will not
automatically reattempt the journal change request. Manually
resubmit the change request by pressing F8=Submit Change
Processor when the object is no longer locked.
6. Press F7=Toggle until the Completed Change Requests view is displayed.
7. Press F20=Clear Listed Requests to remove the objects from the display.
Apply Process - Work with Apply Jobs (3.4 - Target)
• Fast path: Enter 3.4 on any iTERA HA command line on the target
machine.

Apply Process - Work with Apply Jobs (3.4 - Target)
• OR – Fast path 1.1 then F16, then F9.

Select option 4 – Work with Apply Jobs.
Once data has been journaled on the primary node and sent to the target node
via remote journaling, the data is ‘applied’ to the necessary files (i.e.,
transactions are applied to the necessary objects) on the target node via the
apply jobs process.
The Apply process runs only on the target machine. One apply process is
created for each journal (both iTERA HA and User journals).
IMPORTANT
Great care should be taken when overriding the sequence number (option
12). While the instructions to perform an override are documented as
part of the various procedures that require it, Vision Solutions does NOT
recommend you perform this procedure without having first consulted
with CustomerCare.
NEVER ATTEMPT TO OVERRIDE THE SEQUENCE NUMBER

IF THE PRIMARY SYSTEM IS NOT AVAILABLE.
If this procedure is executed incorrectly or inappropriately, there is a

strong possibility that you will have lost the transactions!
Journal Apply Statistics (1.1 F9 or 3.6, Target)

The Journal Apply Statistics screen is accessed on the target node either via 1.1
F9 or 3.6 and is used primarily to determine specific apply job activity. If the
System Monitor indicates there are objects requesting sync, press F9 from that
screen to review the number of unapplied entries as well as the estimated time
it will take for the system to catch up.


Replication Basics
4
Replication Overview
Library and object replication in iTERA HA uses journaling to record
changes to selected objects (files, data areas, data queues) on the primary
node. Journal changes are sent to remote journals on the target. An apply job
runs on the target node and monitors the journal receivers (which are
attached to the remote journals) for new journal entries, then extracts and
writes the data changes recorded in the journal entries to the appropriate
objects on the target node. For objects that are already being journaled by
other journals, iTERA HA simply uses the existing journals to mirror data
for those objects.
Libraries That Should Not Be Replicated

The following libraries should not be replicated in iTERA HA. However,
there may be exceptions. Work with your implementation specialist to
ensure your environment is correctly replicated.
• Libraries that begin with ‘Q’

• MPGLIB - Performance Library
• All iTERA HA libraries
• RecoverNow libraries (if using that product)
• Refer to the iTERA HA v6.0 Reference Guide for a list of objects, object
replication changes, and system components not supported in iTERA
HA.
Essentials Needed for Replication

Before you can begin replication of libraries and the objects within those
libraries, there are several prerequisites that must be met.
• In order for libraries to be replicated to the target node with ownership

and authority intact, user profiles must first be replicated.

Chapter 4: Replication Basics
• In order for user profiles to be replicated to the target box, there are certain
non-mirrored objects that must first be replicated.
• These objects include message queues, out queues, jobds, authorization

lists, etc. These objects are usually found in IBM Q libraries that are not
mirrored to the target node.
IMPORTANT
Vision Solutions does not support and strongly discourages
replication of system libraries beginning with Q, particularly
QUSRSYS. This OS library contains OS-specific objects and other
objects unique to the machine.
Replicating this library could cause OS and/or hardware failures!
If you need to replicate objects from QGPL or QUSRSYS, we

strongly recommend that you use the Non-Mirrored Library Object
Sync, as indicated below. If QGPL is replicated, all E2*, IPT7*,
ITHA*, and IPL* files must be omitted.
If you must mirror objects from this library, do so solely at your own
risk. Work with your implementation specialist or contact
CustomerCare for additional information.
Non-Mirrored Objects (4.30)

From this screen, you will sync output queues, authorization lists, message
queues, etc.
• From the Main Menu, select option 4 – Library Replication Menu. Select
option 30 – Non-Mirrored Sync definitions.

Set Up Non-Mirrored Library Object Syncing

1. Select menu option 10.50, Parameters.
a. Position to data area E2CRTNMLIB.

b. Select option 2=Change.
c. Type YES, then press Enter.
2. Select menu 4.30.
3. Press F16=Add System Default Objects to automatically load the list of

recommended non-mirrored objects and exclusions. The following screen
is displayed:
The following table indicates the sync criteria and exclusions that have
been added to the screen:
Library Object Type Exclusion
QGPL *ALL *CLS
QGPL *ALL *JOBQ
QGPL *ALL *MSGQ Q*
QGPL *ALL *OUTQ Q*
QGPL *ALL *SBSD
QSYS *ALL *AUTL Q*
QUSRSYS *ALL *CLS
QUSRSYS *ALL *JOBQ
QUSRSYS *ALL *MSGQ Q*
QUSRSYS *ALL *OUTQ Q*
QUSRSYS *ALL *SBSD

IMPORTANT
This step is critical for all Q libraries!
4. Manually add an exclusion for the following:
Library Object Type Exclusion
QGPL *ALL *JOBD QDFTJOBD
a. Press F6=Add Sync Criteria.

b. Enter the appropriate data into the Library, Object and Type fields.
c. Press Enter. The screen will close automatically.

d. Select option 5=Work with Exclusions. Enter QDFTJOBD in the
Object column. Press Enter.
IMPORTANT
Other non-mirrored objects in your system may need to be synced.
Evaluate your systems in order to determine if objects in Q libraries
should be replicated. Use the instructions above to add them.

5. Press F9=Sync NM-Library. Enter *ALL for Library and *ALL for Target
System.
6. Press Enter to initiate the sync.
User Profile Replication (5.1)

This option is used to configure maps and execute the process of replicating
user profiles, as well as to apply changes to existing profiles, from the primary
to the target.
Once user profiles have been replicated to the target, any changes to existing
profiles will be automatically replicated.
User profiles must exist on the target machine before libraries can be replicated.
Identical mapping (F16) is required on all nodes. If more than one target node
is defined, F8 is used to copy maps to other nodes.
NOTE
If you have more than one CRG defined, each user profile should be
replicated only through one CRG.
Replicate User Profiles

1. Access the User Profile Replication screen by entering 5.1 on any iTERA
HA command line on the primary node.
NOTE
User Profile Replication should already be enabled in the Replication
Options screen (30.23). If, for some reason, the menu option for
User Profile Replication is not displayed in the Non-Library
Replication screen (5), or you cannot access the screen, access the
Replication Options screen (30.23) then select option 6=Enable for
the *USRPRF component.

2. If more than one target node is defined, the following screen is displayed.
Select the BACKUP 1 node (not the REPLICATE node), then press Enter.
3. The following screen is displayed.
4. Press F18=Submit Info Build to view all profiles on the primary system.
5. Select F16=Dft Map and review the pre-defined profile maps on the
primary node.
Several maps are displayed. These maps control the behavior of the
affected user profiles on the target node.

• The *ALL generic entry will ensure all user profiles not handled by any
other entries (generic or specific) are replicated (Add, Delete, Change,
and Password parameters are all set to Yes). Also, the Disable parameter
should be set to Yes for this entry so that general users cannot access the
target node.
• The other pre-defined entries include the E2*, E2CSTMGR, iTERA
HA Admin profiles, ITERADIRE, ITERAOUTQ, ITERAOWNER,
ITERAUSPF, and Q*. The defaults settings for these entries should be
retained (these profiles should not be disabled on the target).
• iTERA HA will automatically exclude any profiles that begin with the
letter Q. The Q* map entry covers all profiles beginning with the letter
Q (system profiles). However, if you have individual users whose
profiles begin the letter Q, a map must be created for each, so that they
are included.
a. Press F6=Add to add a new entry to the map. The following screen
is displayed:
b. The screen below indicates how the map entry should look for user
profile “Quick”.
c. Enter the entire profile name in the User Profile or Generic* field.
d. Set the values for Add, Delete, Change, Update, and Disable Remote
Profile to Y, unless this profile should be allowed to generally access
the target node (not recommended).

e. Enter an appropriate description for the profile map in the

Description field.
f. Press Enter.
g. Repeat step a through step f to create map entries for any other user
profile not already covered by an existing map entry.
h. Create a map entry for your own user profile, as well as those of any
other system administrators, and make certain to set the Disable
Remote Profile parameter to N so that these profiles will not be
locked out from being able to sign on to the target node.
i. Press F3=Exit when finished.
6. If more than one target node is defined, select F8=Copy Maps. A screen
similar to the following is displayed.
7. Enter option 1=Select then press Enter to copy the map from the Backup 1
node indicated to the Replicate node.
8. Press F3=Exit to return to the main User Profile Search screen.

9. Select F20=Control. The following screen is displayed.
.
10. The defaults should generally be left as defined (if needed, consult F1 Help
for information on these fields). However, the setting for Transport Choice
should be considered as follows:

• If user profiles are managed exclusively via a 5250 emulation program

(green screen), select option E=Exit Point. This indicates that the exit
program will exclusively control User Profile Replication.
• If both the iSeries Navigator and green screen are used to manage user
profiles, select option B=Both. This indicates that both the exit
program and the iTERA HA’s Object Monitor will monitor user
profiles.
• If iSeries Navigator is used exclusively for user profile maintenance
(very rare), select option O=Object Monitor. In this case, the iTERA
HA Object Monitor jobs will monitor iSeries Navigator for changes to
user profiles.
11. Press F7=Add Exit Program to load the exit program with the desired
settings, then F3 to exit.
12. Press F21=Quick Sync to replicate all profiles from the primary node to
the target node (identified in the Target System screen heading).
13. User Profile audits will be automatically executed from the Audit
Command Console on a regular basis (you’ll learn more about this in a
later chapter). They may also be run manually from within the 5.1 screen
as follows:
Audit Description How to Execute
This audits all profiles to On the primary, select 5.1,

verify that they match the F10, option 1 - Audit All.
settings defined in the User
Profile Mapping
Audit All
Maintenance screen (F16).
A profile that differs from
the map settings will be
automatically corrected.
Compares each node to On the primary, select 5.1,

look for profiles that exist F10, option 2 - Existence
on one node but not the Audit; press Enter to
Existence Audit
other. display the User Profile
Existence screen. Press
F18=Submit Audit.
IMPORTANT
Do not start replicating libraries until all profiles have finished
replicating.

NOTE
If profiles have been replicating and a new map is added, then the
profiles associated with the map will need to be audited or resynced.
Object Lock Analysis and Filters

It is very important to identify potential problems before syncing any libraries.
Objects that have exclusive locks on them pose problems in replication.
Because these objects are exclusively locked by another job, the iTERA HA
processes cannot allocate the object long enough to either turn on journaling
or to save the objects to get them over to the target node.
Lock Analysis
1. Select menu option 4.22 on any iTERA HA command line.
2. Press F18=Submit Analyzer,
3. Press F8=All Non-Exempt Libraries.
NOTE
The lock analysis may take a significant amount of time to complete.
After the lock analysis has completed, the screen will display all objects that are
currently locked. This screen should be reviewed on a weekly basis and objects
that are no longer locked (i.e., the “State” of the object is no longer *EXCL or
*EXCLRD) should be synced using option 9=Sync Object.

Once the object has been synced, the object may be removed from this list
using option 4=Delete Lock Record.
Prior to a role swap, you will review this screen after the systems have been
quiesced (and all locks have been released) and then sync all objects that have
not been synced.
7=Toggle Perm Lock is used to retain objects on the screen for review and
syncing prior to a role swap. Objects marked for perm lock are not removed
from the screen when the F18=Submit *EXCL Lock Search is performed.
F10=Sync Roll Request performs a non-mirrored sync of the objects only. It

does not start journaling on the objects, so changes to the objects will not be
replicated. This is typically used for objects that are heavily used. After syncing
the object, check E2MSGLOG on the target for MSG ID HAE0339, which
indicates that the non-mirrored object sync was successful.
Object Sync Filter

The purpose of the Object Sync Filter is to prevent certain objects from being
replicated to the target node. It is important to identify any objects you do not
want to sync to the target system prior to syncing libraries. Examples of objects
to exclude may include files or data areas that may hold license keys, or work
files that are repeatedly being populated and cleared.
The table under the “Omit Filter Types” heading on page 60 illustrates the five
key omit types and illustrates the outcome of what will occur to the object
when the filter is selected. The iTERA HA Reference Guide contains
additional detail on filter behavior.
• Fast path: Enter 4.20 on any iTERA HA command line
• From the Main Menu select option 4 – Library Replication Menu.

Select option 20 – Work with Filters.
• Also accessible from 4.11, F16.

IMPORTANT
Journaling is automatically ended for journalable objects in mirror
journals that match a filter definition. iTERA HA does not start or
end journaling on any object being journaled in a user journal (the
object is being journaled via user journals for some other purpose),
therefore, filters cannot be enforced for those objects. Do not create
filters for objects in user journals.
IMPORTANT
Objects within libraries that have QDFTJRN defined cannot be
filtered. Use the following command to determine whether
QDFTJRN is specified for a library:
WRKOBJ [LibName]/QDFTJRN
IMPORTANT
Objects in mirrored libraries that reside on the target machine that do
not reside on the primary machine and have an Omit Type code of 1
or 2 will be deleted through the audit process. The data area
E2AUDDLT must also be set to Y.
IMPORTANT
The following omit filter is NOT allowed:
Library = *ALL
Object = *ALL
Type = *ALL
Attribute = *ALL
Filter Type = Omit
Omit Type = 2=Omit All
IMPORTANT
Do NOT define a CLRDTA omit filter for:
Object =*ALL
Type = *FILE
Attribute = *ALL
Omit Type = 4
These combined filter settings will clear all files, including DDM
files. If the DDM file is pointing to a file on another system, the files
on the other system will be cleared. This includes the Primary. When
setting up a CLRDTA filter, an acceptable setting for Attribute is PF.

NOTE
During a Virtual Role Swap, changes made to objects that have been
filtered using an IGNCHG filter will be retained.
IMPORTANT
Vision Solutions does not support and strongly discourages
replication of system libraries beginning with Q, particularly
QUSRSYS. This OS library contains OS-specific objects and other
objects unique to the machine. However, we understand that some
third-party application vendors often place objects in this library that
are required on the target systems. The Non-mirrored Library Object
Sync (4.30) should be used for needed objects that exist in QGPL or
QUSRSYS libraries. Check with your third-party application vendor
to identify required objects.
If you must mirror objects from Q libraries, do so solely at your own

risk. Refer to additional information in the iTERA HA Advanced
Features Guide.
Replicating Q libraries could cause OS and/or hardware failures!
IMPORTANT
Under no circumstances should a filter entry exist for the following:
Library = QSYS
Object = *ALL
Type = *LIB
Filter Type = Include
The existence of this filter (in combination with other factors, such
as performing a product upgrade) has resulted in the need to reinstall
the product.
IMPORTANT
Do NOT define an Include filter for:
Library = QSYS
Object = *ALL
Type = *FILE

Omit Filter Types
Clear the
Delete Create the
Copy the Journal contents of
object from object definition
entire the object the object on
Omit Type target node
object to
on the target
on the target node
Code if it already
the target
and update
primary (keep only
exists definition when
there? node? changes occur? node? object
definition)?
1 = RMTDLT Yes* No No No No
2 = OMTALL No No No No No
3 = IGNCHG No Yes** No No No
4 = CLRDTA No Yes Yes*** No Yes****
5 = NOJRN No Yes Yes*** No No
* Deleted during the CHKOBJMTCH audit and only if the data area E2AUDDLT is set to Y.
** Library sync only
*** By default, filtered objects are not journaled. Therefore, changes to the objects can only be tracked
through QAUDJRN, which is processed by the Object Monitor. Changes detected by the Object Monitor
will result in a resync of the objects.
**** Content clearing applies only to physical files and save files.
How to Create an Object Filter Entry

1. Select F6=Add Filter.
2. Fill in the display with the appropriate data. Consult the table under the
“Omit Filter Types” heading on page 60 for descriptions of the filter types
available. Consult the iTERA HA v6.0 Reference Guide for more
information on other fields in this window.

In the following example, all files in library “MIKER” will be filtered from
replication to the target node.
3. Press Enter to add the filter.
Filter Objects From User Journals

The following procedure should be used for objects in user journals that should
be excluded from replication.
1. Select 4.11.
2. Select option 7=Object Detail.
3. Select option 8=Update Status.
4. Enter an X in both the Journal Status and Syncing field, then press Enter.
The display is returned to the Mirrored Object Maintenance scree. An X is
displayed in both the Jrn and Snc fields.
5. Repeat for all objects that you do not want journaled.

Sync Defined Non-Mirrored Objects By Library

(4.31 - SYNCNMLIB)
From this screen, the non-mirrored library sync requests built previously in the
Non-Mirrored Sync Definitions (4.30) screen are processed. 4.30 F9=Sync
NM-Library and 4.31 both execute the SYNCNMLIB command.
Before you use this menu option, the library must exist on the target (it can be
empty, but it must exist there).
If you have more than one target system defined, you may press F4 to prompt
the list of systems defined as targets (and/or replicate nodes).

Select option 31 – Sync Defined N/M Objects by Library –
SYNCNMLIB.
Sync Non-Mirrored Objects (4.32 - SYNCOBJ)

This option allows you to designate non-mirrored objects within a library for
synchronization to the target node. Normally, this option is used when you
want to copy objects to the target node that seldom change on the production
node; typically, objects in IBM system libraries.
Objects that are selected for synchronization through this screen will be copied
to the target node, but will not be monitored for changes or resynced by iTERA
HA. If you want to resynchronize the objects, you must use this screen again.
Objects can be included or excluded for a library by type of object, by object
name or using wildcard naming conventions.

Import Information From SAT Lite
IMPORTANT
NMO sync is not intended to ensure that libraries defined in this
process are in sync between the primary and the target. It is intended
only to save objects on the primary and restore them to the target. It
will NOT clean up obsolete objects on the target machine.

Select option 32 - Sync Non-Mirrored Object – SYNCOBJ.
NOTE
Setting the Save Definition field to *YES will add the object to the
Non-Mirrored Library Object Sync screen (4.30) so that the item
will be monitored for changes and resynced by iTERA HA when
necessary.
NOTE
The library entered must exist on the target node in order for this
option to work.
Import Information From SAT Lite

The Vision Solutions product SAT Lite, is an analysis tool that gathers
necessary information about your system for replicating objects and libraries.
As part of the iTERA HA pre-installation process, you were instructed to run
SAT Lite.
The results of the analysis will help you identify and understand important
information about your system in relation to syncing using iTERA HA and
will help you identify the libraries and objects that should be selected for
replication. The list of libraries produced by the SAT Lite analysis can be

imported into the iTERA HA All Available Libraries list (4.11). This will save
time by not having to run the Library Analyzer (4.11, F18).
Import the list of libraries from SAT Lite into iTERA HA by selecting menu
option 10.3. (This option runs as soon as it is executed. No screen is
displayed.)
Library Replication
In order for iTERA HA to be able to track and maintain objects being
replicated, a number of events must take place:
• The library must be identified and associated with a journal. (The journal
should already be created and all mirroring functions working properly.)
• Objects within the library must be identified and associated with a journal.
• Journaling of objects within the library must be started.

• The library must be synced to the target node using either net sync or tape
sync.
• Once the library has been sent across the network or is on tape, it must be
restored on the target node. (At the time of restore on the target, the
objects will also start journaling on the target node.)
Work with Libraries

Select option 11 – Work with Libraries.
Selected fields are described in the table below the screen. For additional field
descriptions, consult the iTERA HA v6.0 Reference Guide.

Library Procedures
Option/Field/Function Description
If there are libraries you do not wish to sync, you may

16=Clear New Status remove them from the New Libraries view (F7) by
selecting this option.
Use this option to Quick Sync libraries. You may

21=Quick NetSync
select multiple libraries before pressing enter.
To completely stop replication for a library, execute

the following steps:
14=Cancel Syncing (Check E2SBS after each step.)
4=End Journaling 1. Cancel syncing (Option 14, primary)
24=Remove CRG
Assignment 2. End Journaling (Option 4, primary)
3. End Journaling (Option 4, target)
4. Remove CRG Assignment (Option 24, primary)
Be aware of the size of the libraries to be synced. You

Size (MB)
may want to sync larger libraries via tape.
Use F7 to toggle the view between Mirrored Libraries

F7=Toggle View Only, All Available Libraries, New Libraries Only, and
All Libraries.
Displays the Object Sync Filter screen, where you can

F16=Filters
define specific objects to omit from syncing.
NOTE
There is a known issue in relation to joined logical file authorities.
Joined logical file authority cannot be replicated exactly the same to
the target system. This is an OS limitation.
When a joined logical is created on a system, the initial user

(typically the one who creates the file) gets *ALL object authority,
even though the user really can’t use all of the authority. Records in
joined files cannot be added, updated or deleted directly, even
though the authority says you can. For additional information, refer
to the topic under 4.11 in the iTERA HA Reference Guide.
Library Procedures
The following procedures pertain to the mirroring and management of
libraries. Syncing libraries via a network in iTERA HA is a simple and
commonly used procedure, requiring just a few steps. However, because of the
length of time that it can take to complete a tape sync or net sync of a large
library, issues can arise with these types of syncs. Normal iTERA HA processes
(such as auditing and journal receiver deletion) must be considered when
attempting these procedures. Tape syncs where the system is located at an

off-site facility can take from a few hours to several days. In this situation, the
journal receivers must be prevented from being deleted by the journal manager.
Net syncs for large libraries are dependent on a number of factors, including
object size, system bandwidth, etc. Additionally, files in a library that have
logicals with unique key constraints in a different library will require special
handling.
• For normal-sized library syncs via the network, if the library has not
previously been mirrored, begin with the procedure “Specify a new library
for mirroring” on page 67, then sync the library using the procedure
“Network Sync a Library” on page 68.
• For large library syncs and all library tape syncs, if the library has not
previously been mirrored do the following:
– Begin with the procedure “Specify a new library for mirroring” on

page 67
– Continue with the procedure “Steps to perform prior to a sync or
resync” on page 66
– Sync the library using either “Network Sync a Library” on page 68, or
“Tape Sync a Library” on page 69
– Complete the process using “Steps to perform after a sync or resync” on
page 70.
How determine whether to use the Large Library

Syncing Procedures
Because a number of variables unique to your environment affect the speed at
which iTERA HA is able to sync and restore libraries, criteria for defining a
large library cannot be exactly defined. As a rule of thumb, any library that
takes longer than an hour to sync via the network should be considered large.
Therefore, Vision Solutions’ recommendation is to use the procedures “Steps
to perform prior to a sync or resync” on page 66 and “Steps to perform after a
sync or resync” on page 70 when in doubt about your systems’ ability to
accommodate library synchronization. Use these procedures when at least one
object in the library to be synced has more than seventeen million transactions
in a twenty-four hour period.
Steps to perform prior to a sync or resync

IMPORTANT
Do not end the iTERA HA subsystem during the resync and restore
process.

Library Procedures
The following steps should be performed prior to executing a sync or resync:
1. Place the Roleswap Monitor job on hold. In E2SBS, select option 3 for the
xx_RSRMON job.
2. End the Audit Command Console on all nodes (1.8, F11=End Auto
Audit).
3. From the Job Scheduler (WRKJOBSCDE), place all the iTERA HA

audits (E2*) on hold using option 3.
4. Place the Journal Manager (XPJRNMGT) job on hold on both systems

(3.33, option 3 on the job).
5. Hold the sync job (xx_SNC_yyy) on the Primary system (2.11 or E2SBS).
6. If syncing a new library, perform the steps in“Specify a new library for
mirroring” on page 67.
7. Perform the tape sync or large net sync indicated in “Network Sync a
Library” on page 68 or “Tape Sync a Library” on page 69.
8. Perform the steps indicated in the section “Steps to perform after a sync or
resync” on page 70.
Specify a new library for mirroring

1. Select menu 4.11, Work with Libraries, on the Primary.
2. Determine the default journal. If the journal has not been defined in
iTERA HA, define it by selecting F15=Change Default Journal.
In order to use additional journal(s) for selected objects they will need to
be created and/or the journal(s) defined to iTERA HA.
Do the following steps for each of the journals to be used:
– To create a new journal and add the new journal for this library, see
“Create A New Mirror Journal (3.1)” on page 35.
– To use a User journal that is not defined to iTERA HA, see “Define a
User Journal to iTERA HA” on page 37.
NOTE
Only mirror journals are available for selection. A user journal can
only be assigned as the default journal if it is changed to a mirror
journal (3.1 opt 8). Once a library is synced, the type of journal
should generally be changed back to USER (3.1, option 2; Jrn Type
parm). Otherwise, if journaling is ended for the selected library,
journaling will be ended for all objects assigned to the user journal.

3. If needed, use the 4.20 (Object Sync Filter) screen on the Primary node to
set up filters for objects.
NOTE
If objects need to be filtered from replication for this library, use
F16=Filters to define the filter. (Filters can only be defined for
objects that are not already being journaled to user journals. For
more information on filters, see “Object Sync Filter” on page 57.)
4. From the Work with Libraries screen, select 4.11, F7, primary, to display
the list of all available libraries. The screen heading will indicate “All
Available Libraries”. Use option 1=Select to select the desired library, then
press Enter. Press F7=Toggle View until the “Mirrored Libraries Only”
view is displayed.
5. If the assigned mirror journal shown for the library is not correct:
a. Select option 8=Change Journal; press Enter.

b. Select option 1=Select with Default Image; press Enter.
c. Review the assigned mirror journal and verify that it is correct.
6. If objects in the library need to be assigned to a different journal, see “Sync
a Library That Needs Objects Changed to a Different Journal Than the
One Assigned to the Library.” on page 71.
7. Sync libraries using either network or tape.

• There are preparatory steps that must be performed prior to and after a
tape sync or large net sync. See “Steps to perform prior to a sync or
resync” on page 66.
• To sync a library via the network, follow the instructions in “Network
Sync a Library” on page 68.
• To sync a library via tape, follow the instructions in “Tape Sync a
Library” on page 69.
• After performing the tape or large net sync, also execute the steps in
“Steps to perform after a sync or resync” on page 70.
Network Sync a Library

NOTE
“Steps to perform prior to a sync or resync” on page 66 should be
completed prior to executing this procedure.

Library Procedures
IMPORTANT
process.
2. For resyncs only, locate the affected library; if the status is Active, select opt
14=Cancel Syncing, press enter.
3. Select option 21=Quick NetSync for the library you wish to sync, then
press Enter. Quick syncing assigns the library to the default journal, starts
journaling on the objects, saves the library and restores it to the target
node. The library’s status will change to QuickSnc and then to Active once
the journaling step has completed.
NOTE
Multiple libraries can be selected simultaneously, but be aware of
available disk space bandwidth, and CPU resources.
4. Press F6=Work with library sync. The Library Syncing Status screen is
displayed. The Status column indicates where it is in the process of saving,
sending, and restoring the library to the target. Once the library has been
restored, the status will indicate Syncing.
5. Complete the steps in the section “Steps to perform after a sync or resync”
on page 70.
Tape Sync a Library

More than one library at a time can be synced to tape as long as all libraries fit
on one tape. Prepare the libraries then select all at the same time.
NOTE
“Steps to perform prior to a sync or resync” on page 66 should be
completed prior to executing this procedure.
IMPORTANT
process.

1. Select option 23=Quick Journal. (This assigns the library to the default
journal then starts journaling on all objects.) The status will change from
QuikJrn to Active.
2. Select F6=Work With Library Sync. Press F5 until the status on the left
indicates Ready. Do not continue until after the status for each selected
library has changed to Ready. (This may take some time.)
3. Select option 6=Select to Sync for the libraries then press Enter.
4. Select F18=TapeSync Selected Libraries. Enter the Tape Device and
Volume ID you want to use for the tape sync. (A message may be received
in QSYSOPR instructing you to load the tape. Select option G to go.) A
job will be submitted to the job queue which will finish saving the library
to tape.
5. The sync status in the 4.12 screen will change from Prepping, to Saving, to
On Tape when finished.
6. Transport the tape to the target system.
7. Mount the tape on the target system.
8. On the target machine select Restore LibSync from Tape (4.51). A window
will prompt for the name of the tape drive where you loaded the tape.
Enter the tape device and Volume ID, press Enter. The system will start to
read the tape when it is needed. This may take from a few minutes to a few
hours. (A message may be displayed in QSYSOPR instructing you to load
the tape. Select option G to go.) A job will be submitted to the job queue
which will finish restoring the library to the target system.
9. After the sync has completed, on the primary, select Library Syncing Status
(4.12) and verify that the library has been identified and is syncing (status
should indicate SYNCING).
10. Continue with the steps the section “Steps to perform after a sync or
resync.”
Steps to perform after a sync or resync

The following steps are required after performing a sync or resync, and after
ensuring the library is syncing normally.
1. On the primary, select Library Syncing Status (4.12) and verify that the
library has been identified and is syncing (status should indicate
SYNCING).
2. In the iTERA HA subsystem release the SYSMON job and the
xx_RSRMON jobs (E2SBS, opt 6).

Library Procedures
3. Start the Audit Command Console on both nodes (1.8, F11=Start

Auto Audit). If audit jobs have been held in the job scheduler, release
them.
4. Release the XPJRNMGT job on both systems (3.33, opt 6 on the job).
5. Release the Sync job (xx_SNC_yyy) in the subsystem on the primary
node (E2SBS, opt 6).
Sync a Library That Needs Objects Changed to a

Different Journal Than the One Assigned to the
Library.
This procedure may be used when you want to assign one file in a library to a
journal and another file in the same library to a different journal before you
start synchronizing the entire library.
If needed, this step should be done before you initially sync the library because
you are pre-assigning objects to journals.
If the library is already actively journaling, you can reassign individual objects
from the Mirrored Object Maintenance screen (4.21, option 7=Transfer
Journal). An exclusive lock on the object is required.
1. In the Work with Libraries (4.11) screen, verify that the journal you want
to use is defined as the default journal. If not, change the default journal to
the one you want to use by selecting F15=Change Default Journal.
2. Use option 1=Select on the desired library.
3. Use option 7=Object Detail on the library to display the Journaled Object
Maintenance screen.
NOTE
Option 7 will display the Journaled Object Maintenance screen only
if the library’s status is Inactive. Otherwise, the Mirrored Object
Maintenance screen (4.21) is displayed.
4. If no objects are displayed, press F18=Submit Build to load.
5. Once objects are loaded, use the Position field to locate the object you want
to change to a different journal.
6. Select option 2=Change Journal to change the journal. A list of available

journals is displayed, to which you can assign the object. Select the journal
using option 1=Select With Default Image.
7. Once all desired objects have been changed, exit this screen by selecting
F3=Exit.

8. Sync the library via network or tape. See “Network Sync a Library” on
page 68 or “Tape Sync a Library” on page 69.
Remove a Library From Being Mirrored by iTERA

HA
1. Select the Work with Libraries menu (4.11, all nodes).
2. On the primary, select option 14=Cancel Syncing. Press enter. Read the
screen message then answer with a Y. Press Enter.
3. Select option 4=End Journaling on the library, then press enter. This
option will submit a job that will try to end journaling for any object that
is journaled to a mirrored journal. The system will not end journaling if
the journal is not a mirrored journal. You will not be able to proceed until
the job finishes (all nodes).
4. Check E2MSGLOG for messages. Look for HAE0118, which indicates

the number of objects for which journaling was ended and the number
that were locked. If objects are locked, end the locks then reattempt the
previous step.
5. Locate the same library on the target node. Select option 4=End
Journaling, then press Enter. This will only end journaling for objects
being journaled to a mirrored journal.
6. Check E2MSGLOG for messages. Look for HAE0118 which indicates the
number of objects for which journaling was ended and the number that
were locked. If objects are locked, end the locks then reattempt the
previous step.
7. On the primary, select opt 24=Remove CRG Assignment and press Enter.
This will clear any entry in the Status column and removes it from the
Mirrored Libraries view (the library will still be displayed in the All
Available Libraries view).
8. Optional: On the target, delete the library by using the DLTLIB

command.
If a journal no longer has objects attached to it, you can remove the journal
from iTERA HA. Refer to the iTERA HA v6.0 Advanced Features Guide for
the procedure “Remove a journal from being defined in iTERA HA”.

Library Procedures
Change the Mirror Journal Assigned to the Library

Use this procedure to change the journal for any objects that will be added to
the library in the future.
IMPORTANT
The journal you plan on using must be defined to iTERA HA before
it can be assigned.
- If you need to create a new journal see,“Create A New Mirror
Journal (3.1)” on page 35.
- If you want to add a user journal to iTERA HA see, “Define a User
Journal to iTERA HA” on page 37.
1. Select the Work with Libraries menu (4.11, primary).
2. Select option 8=Change Journal. Press Enter.
3. Select the journal with option 1=Select with Default Image.
NOTE
This action affects only non-journaled objects added to the library
after the change.
Change the Journal to Which a Library and All Its

Objects is Being Journaled
Use this procedure to change the journal for existing objects in the library.
IMPORTANT
This process will end journaling on the selected objects and start
journaling to a different journal. You can not end journaling if any
process is using the object.
1. Select the Work with Libraries menu (4.11, primary).
2. Select option 8=Change Journal. Press Enter.
4. Select the Journaled Object Change Requests menu (3.22 primary).
5. Select F18=Import Object Change Requests.
a. Library: Enter the name of the library.

b. Object: Enter the object name (*ALL for all objects).
c. Type: Enter the type of object (use IBM definitions *FILE, *DTAARA,
etc).

d. Press Enter.
7. Either press F9 to submit the request immediately or submit the program

E21080CPS at a time when the objects will not be locked.
Replicate a Library to an ASP Other Than ASP 1.

The following procedure is used to change the journal assignment for a library
to a different ASP.
1. If the library is currently being mirrored, first follow the procedure

“Remove a Library From Being Mirrored by iTERA HA” on page 72 then
return and continue the instructions.
2. Save the library to device and restore to new ASP assignment. (The new
ASP is number 2 for illustration purposes.) Note the Restore to ASP number
parameter below:
Restore Library (RSTLIB)
Type choices, press Enter.
Restore to library . . . . . . *SAVLIB Name, *SAVLIB
Restore to ASP device . . . . . *SAVASPDEV Name, *SAVASPDEV
Restore to ASP number . . . . . 2 1-32, *SAVASP
Output . . . . . . . . . . . . *NONE *NONE, *PRINT, *OUTFILE
File to receive output . . . . Name
Library . . . . . . . . . . . . *LIBL Name, *LIBL, *CURLIB
3. Create the library on target node for the desired ASP so the remote journal
jobs can use it.
4. Create the journal receiver and the user journal for the library in the
desired ASP in the same library using the following commands:
CRTJRNRCV JRNRCV(LIBX/RCVASP) ASP(2) TEXT('Receivers
for ASP2 Journal')
CRTJRN JRN(LIBX/JRNASP2) JRNRCV(LIBX/RCVASP2) ASP(2)
5. Load new user journal in 3.1 Local Journal Maintenance, F8=Load New
User Journal.

Library Procedures
6. Change the user journal to a mirror journal to allow it to be used as a

default journal for the syncing process. Use 3.1 on the primary, option
2=Change Group & Description, change the journal to a Mirror journal.
7. Verify that the journal has the mirror process set to “On”. If it is not, use
option 32=Toggle Mirror Status to enable the mirror status. Once it is on,
verify that remote journals have been created using the 3.3 screen.
8. Verify on the target machine that there is an apply job created. If any of the
components are missing, return to 3.1 on the primary machine and
rebuild the components.
9. Verify again that the remote journals and the apply jobs are in place.
10. Select 3.4, Apply Job Maintenance, and start the apply process on the
target machine as follows:
a. Select option 11=Suspend,

b. Select option 12=Override Next Seq
c. Select option 2=Use the last seq of the current attached receiver, then
Press Enter.
d. For the parameter Delete all prior Journal Receivers form primary, select
option N, enter a reason in the Reason for Change line, then press Enter.
e. Wait for the apply status to become active. If it does not, suspend and
do the override step again.
f. Once it becomes active, select option 14=Restart Job to restart the
apply job.
11. On the primary, enroll the new journal into the Journal Manger by using
3.32, F6=Add.
12. From 4.11 on the primary machine, change the default journal using the
F15=Change Default Journal. Select option 1=Select with Default Image
on the journal that you created in the other ASP.
13. Verify that the journal now shows as the default journal on the 4.11 (Work
with Libraries) screen.

14. Resync the library by using the instructions in “Network Sync a Library”
or “Tape Sync a Library” on page 69.
Object Procedures
Syncing objects via a network in iTERA HA is a simple and commonly used
procedure, requiring just a few steps. However, because of the length of time
that it can take to complete a tape sync or net sync of a large object, issues can
arise with these types of object syncs. Normal iTERA HA processes (such as
auditing and journal receiver deletion) must be considered when attempting
these procedures. Tape syncs where the system is located at an off-site facility
can take from a few hours to several days. In this situation, the journal receivers
must be prevented from being deleted by the journal manager. Net syncs for
large objects are dependent on a number of factors, including object size,
system bandwidth, etc. Additionally, files that have logicals with unique key
constraints will require special handling.
• For normal-sized object syncs, use the procedure “Resync an Object (or
Objects) Via the Network” on page 77.
• For large object resyncs via network or tape, review “How to Determine
Whether to Use the Large Object Sync Procedure” on page 77, then
execute the procedure “Large Object Resync Via Network or Tape” on
page 79.
• For normal-sized object resyncs via tape, use the procedure “Large Object
Resync Via Network or Tape” on page 79 and ensure the objects are larger
than the value specified in the Max Network Sync Size parameter (1.1,
F6=Objects Requesting Sync, F18=Update Max Network Size).
How to Change the Max Network Sync Size Parameter

The value specified in the Max Network Sync Size parameter in the Objects
Requesting Sync screen (1.1, F6) determines whether objects will be synced via
the network or tape. Objects that are smaller in size than the value specified are
automatically synced via the network. Objects greater than the size specified
are eligible for tape sync.
1. From within Objects Requesting Sync screen (1.1, F6) on the primary,
select F18=Update Max Network Size.
2. Enter a value for Maximum Sync Size in kilobytes.
NOTE
Entering all 9s (999999) in this field will enable all objects to be
synchronized via the network.

Object Procedures
Resync an Object (or Objects) Via the Network

Initiate an object resync from any of the following screens:
• 4.21, option 6=Resync, from any node
• 1.22, option 6=Resync, from the target only
NOTE
If more than one target node is defined, then to resync to all nodes,
initiate the resync from 4.21, opt 6, primary. If the object is to be
resynced to only one of the target nodes then initiate the resync from
that node using either 1.22, opt 6 or 3.7, opt 6.
How to Determine Whether to Use the Large

Object Sync Procedure
Several variables can affect the syncing process. In addition to the manual
resync requests described in the object resync procedure described above,
various programs can initiate resyncs, including the following:
• When a RGZPF is performed on the primary and the target is unable to

replay it, it will automatically request a resync.
• Audits that detect object differences will also sometimes be automatically

resynced. These objects will be displayed in 1.1 F6.
• The Heal process will automatically request a resync when it cannot

correctly heal a record.
Because a number of variables unique to your environment affect the speed at

which iTERA HA is able to sync and restore objects, criteria for defining a
large object cannot be exactly defined. As a rule of thumb, any object that takes
longer than an hour to sync via the network should be considered a large
object. Therefore, Vision Solutions’ recommendation is to use the “Large
Object Resync Via Network or Tape” procedure when in doubt about your
systems’ ability to accommodate object resynchronization. The following are
issues that would warrant using the large object syncing procedure:
• The process that drains data queues within the restore process ends
abnormally. (The data queue will remain in the work library.)
• The object to be synced has more than seventeen million transactions

in a twenty-four hour period.
• After syncing, the object immediately falls out of sync. The reason code
“Bad write” is displayed in 1.1, F6.

• The object repeatedly appears in 1.1, F6.
• Logical files attached to the object being resynced have unique key
constraints. To identify whether a file has logicals with unique key
constraints, do the following:
1. Type the command DSPDBR then press F4 to prompt.
2. Enter the name of a file and library to query, then press Enter. The
output will be displayed on the screen.
3. Position to the bottom of the screen and locate the Files Dependent
On Specified File section. If no dependent files are listed then the
additional steps are not required. If dependent files exist, each
dependent file will need to be checked for unique key constraints.
Note the name of each dependent file listed in the screen output,
then press F3=Exit to return to a command line.
4. Execute the command DSPFD for each dependent file. For example:
DSPFD ITHA/E2POBJL1
Locate the Access Path Description parameter, as shown in the

screen below.

Object Procedures
5. If the setting for Unique key values required is *YES, then execute
the “Large Object Resync Via Network or Tape” procedure,
described below.
Large Object Resync Via Network or Tape

If any of the criteria in the previous section were met, use this procedure to
sync large objects via the network or tape.
NOTE
This procedure also includes an optional step for net syncs for
starting up an additional sync job through which the large object will
be processed. By doing this optional step, other object resyncs that
occur as the large object is being processed will not be held up. The
process entails placing the large object on hold, placing the sync job
on hold, submitting the extra sync job to process the large object,
then releasing the standard sync job.
If the optional procedure is not performed, additional sync jobs can

still manually be started after the large object is resynced. However,
using this technique will require having to watch for new objects
being added to the Objects Requesting Sync screen and then pressing
F16=Submit Extra Network Sync each time an additional sync job is
desired.
1. On all nodes, select 1.7, Role Swap Readiness Monitor. Position to the
OBJSNCSTS test and select option 2=Change. Enter N for the Run All
Tests and Run Test Cmd Test parameters, then press Enter.
2. On all nodes, select 1.8, F11=End Auto Audit, to end the Audit
Command Console.
3. On all nodes, from the Job Scheduler (WRKJOBSCDE), place all the
iTERA HA audits on hold using option 3.
4. On all nodes, select 3.33. Select option 3=Hold for the XPJRNMGT job.

5. On the primary, select 1.1, F6=Objects Requesting Sync. Review the value
defined for the Max Network Sync Size parameter.
• In order for the object to be resynced via tape, it must be larger than
the value defined.
• In order for the object to be resynced via the network, it must be
smaller than the value defined.
If needed, change the value using F18=Update Network Max Size.
6. If the object is highly active (if from the time it is saved to the time it is
restored within the iTERA sync process it exceeds approximately 18
million entries in the journal), do the following:
a. Select 4.21, Work with Mirrored Objects, on the primary.

b. On the highly active object, select option 2=Change Options.
c. Type a Y in the Very Active (SDQ Ofl) field, then press Enter. The
display is automatically returned to the 4.21 screen.
7. In 1.1, F6, check whether the object to be resynced appears in the list. If
the object is not already displayed in 1.1 F6, initiate the resync for the
object using any of the following menu options:
• 4.21, option 6=Resync, from any node

NOTE
If more than one target node is defined, then to resync to all nodes,
initiate the resync from 4.21, opt 6, primary. If the object is to be
resynced to only one of the target nodes then initiate the resync from
that node using either 1.22, opt 6 or 3.7, opt 6.
8. On the primary, select 1.1, F6. If the object is less than the Max Network
Sync Size defined it will be automatically synced via the network. The
syncing process can be monitored from this screen.
NOTE
If the objects are synced via the network AND there are unique key
constraints, then the apply job for the journal being used to journal
the objects must be held before the restore process has completed.
Select 3.4 on the target, then select option 11=Suspend Process State.
(If unsure which apply job to suspend, check the Mirrored Object
Maintenance screen [4.21] for the journal being used to journal the
objects.)

Object Procedures
For network syncs, monitor for additional objects requesting sync. Press
F16=Submit Extra Network Sync to accommodate syncing and to prevent
a bottleneck through a single sync job.
9. Optional: For network syncs only, you may initiate a second sync job so
that the large object resync will run through the second sync job, thus
freeing up the first sync job to process any other sync requests that may
execute during the time the large object is being resynced.
a. In 1.1, F6, select option 3=Hold on the large object.

b. Place the xx_SNC_nn job in the iTERA subsystem on hold (E2SBS,
3=Hold).
c. In 1.1 F6, select option 6=Release on the large object.
d. Press F16=Submit Extra Network Sync to start a second sync job.
(Additional sync jobs are named xxSN2_nn, where nn is the number of
the job.)
e. Release the xx_SNC_nn job in the iTERA subsystem (E2SBS,
6=Release).
f. Monitor the progress of the large object sync from 1.1, F6, as needed.
10. For network resyncs, skip to step 11 on page 82. For tape syncs, if the
object exceeds the Max Network Sync size, perform the tape sync, as
follows:
a. Load an initialized tape into the tape drive and make the tape drive
ready.
b. Press F20=Tape Sync.
c. Select the object(s) to be synced via tape with option 1=Select Object
to Save or F21=Select All.
d. Press F6=Create Tape.
e. Specify the Tape Device and Volume ID and press Enter. Or, if using
an advanced tape system, select F4=Select Volume Set, specify the tape
drive, volume ID, and volume set, then press Enter.
f. Press F9=Submit Tape Sync.
g. A message may be displayed in QSYSOPR, indicating to load the tape.
Select option G to go.
h. A job will be submitted to the job queue which will finish saving the
objects to tape. Once this is finished, the sync status for the objects in
the 1.1 F6 screen will indicate that the objects are on tape.
i. Transport the tape to the target system and mount it.
j. If the objects are synced via tape AND there are unique key constraints,
then immediately suspend the apply job for the journal being used to
journal the objects. Select 3.4 on the target, then select option
11=Suspend Process State. (If unsure which apply job to suspend,
check the Mirrored Object Maintenance screen [4.21] for the journal
being used to journal the objects.)

k. On the target machine select 4.52, Restore Object Sync from Tape. In
the window that is displayed, enter the name of the tape drive where
you loaded the tape. Enter the tape device and Volume ID, press Enter.
The system will start to read the tape when it is needed. This may take
from a few minutes to a few hours.
l. A message may be displayed in QSYSOPR instructing you to load the
tape. Select option G to go.
m. A job will be submitted to the job queue which will finish restoring the
objects to the target system.
n. Select 1.1, F6 on any node. The tape sync is complete when the object
is no longer displayed on the screen.
11. For all object resyncs where there are unique key constraints, immediately
after the sync has completed, do the following. (If there are no unique key
constraints, skip to the next step.)
a. Execute the command EDTRBDAP. Monitor the rebuild of access paths

for the objects with unique key constraints until completed.
b. Start the apply job for the journal being used to journal the objects.
1. Select 3.4 on the target.
2. Select option 13=Activate Process State.
3. Select option 14=Restart Job.
NOTE
Do the remaining steps only after it appears that the sync process has
been successful and the objects have remained in sync for several
hours.
12. On the target, select 4.21, Mirrored Object Maintenance. Position to the
object and verify that the Jrn, Snc, and Omt fields indicate Y, Y, and blank,
respectively. Any other settings will cause the object to be resynced again.
13. On the primary, select 1.7, Role Swap Readiness Monitor. Position to the
OBJSNCSTS test and select option 2=Change. Clear the N for both the
Run All Tests and Run Test Cmd Test parameters, then press Enter.
14. Start the Audit Command Console on all nodes (1.8, F11=Start Auto
Audit, F7=Ignore Time Setup). If audit jobs have been held in the job
scheduler, release them.
15. On all nodes, select 3.33. Select option 6=Release for the XPJRNMGT
job.

Object Procedures
Change the Journal to Which an Object in a

Library is Being Journaled
IMPORTANT
journaling to a different journal. You cannot end journaling if any
process is using the object!
1. Select Mirrored Object Maintenance (4.21, primary).
2. Select option 7=Transfer Journal for the object.
3. Select option 1=Select With Default Image on the journal to be used. Press
Enter.
4. The Library/Journal data in the 4.21 screen will be updated with the new
library and journal.
Change the Journal Image Being Used by the

Object
IMPORTANT
journaling to a different journal. You cannot end journaling if any
process is using the object!
1. Select Journal Change Request (3.22 primary).
2. Select F18=Import Object Change Requests.

3. Enter Library, Object (*ALL for all objects), and Type (use IBM
definitions *FILE, *DTAARA, *DTAQ, etc.). Press Enter.
4. Select the journal image to be used:
– Select option 7=Select After Images, or

– Select option 8=Select Both Images for before and after
Press Enter.
5. Do one of the following:
– Select option 9=Attempt request for each object.

– Press F8=Submit Change Processor to submit the request immediately.
– Submit the program E21080CPS at a time when the objects will not be
locked.

System Library AutoSync (4.10)

The System Library Autosync streamlines the initial syncing of libraries and
objects. It allows you to predefine how you want the system to be
synchronized, and the order in which syncing will occur.
With the System Library Autosync, replication is scheduled and does not
require an operator to be in attendance. Synchronization definitions are
created for objects at either the library or object level. The definitions include a
set of either individual objects, or entire libraries. A definition is assigned a
relative priority status, which is the order in which that definition will be
synced. A low number indicates a high priority. The sync definitions assigned a
higher priority will be synced first.
Sophisticated targeting functionality allows for a wide range of syncing

possibilities ranging from selection of entire libraries down to object by object,
which helps prevent locking issues and makes the replication go much more
smoothly. Object selection masks include generic wildcard functionality, which
expands the ability to select the desired libraries and objects.
When scheduling replication, a time frame is assigned, typically during a time

of relatively low system usage. Further, in order to control the system resources
used during syncing, the number of sync jobs running at any given time can be
controlled.

To set up the Library AutoSync, do the following:
1. Access the System Library AutoSync from menu 4.10 on the primary. Press
F8=Selection to display the Library Selection Definitions screen.
2. Press F6=Add Selection to display the Select Libraries for Auto NetSync
screen.
3. Fill in the screen with the desired data. Instructions for only a few of the
key fields are described below. If additional information is needed, consult
the iTERA HA v6.0 Reference Guide.

Field Instruction/Notes
Selection Set Name Designate a name for this Autosync Definition.
Specify the Selection Range. Wild cards may be used.

Selection Range • From: Selects the first library beginning with the
characters specified.
• Thru: Inclusive of the entry specified on this line.
Provides the ability to include or exclude certain

Selection Mask
libraries from the Autosync definition.
Indicates the maximum file size that will be allowed

Library Max Size
to sync via the network.
The priority in which the AutoSync definition will be

Syncing Priority processed. Designate a lower number for a higher
priority.
*DFT will use the default journal specified. You can

Journal
specify or change the default journal in 4.10, F8, F13.
The usual setting is A (after). However, if using

Record Images
Commitment Control, you must use B (both).
*LIB will do a SAVLIB of the entire library and repli-

cate the library as a single unit.
*OBJ will first create the library on the target side
Library Sync Method then use an audit to sync the contents of the library,
object by object. The benefit of the *OBJ method is it
breaks the library up in to if you can't save an entire
library due to locking issues (e.g., if using commit-
ment control), or if the libraries are excessively large.
Provides the ability to identify the exclusively locked

objects. If Y is specified for Filter Locked Objects, the
Identify *EXCL Objects
sync process will automatically filter the locked
objects.
4. After the AutoSync definition is complete, press Enter. The screen will be
returned to the main AutoSync screen. The Syncing Status will indicate
Defined.

NOTE
At this point, you may choose to immediately begin syncing the
definition or use the System Sync Monitor to schedule syncing for
later. If you want to immediately sync the definition continue with
the next step below. However, if you are using the System Sync
Monitor to initiate and manage replication (recommended), skip to
step 7 on page 88.
5. Select option 8=Submit Sync, to sync without using the monitor.
NOTE
When using this option, the status should indicate “Defined”. Do not
set the status to “Ready” (F10 or option 11). If set to “Ready”, you will
not be able to select option 8, since you make it ready for the
monitor to control it.
6. Select option 8=Submit Sync for the library selection definition. This will
build the list of libraries that meet the selection criteria. The status will
change from Defined to Loading, to Submitted, to Complete.

7. Select option 5=View Results to view the libraries that are defined in the
AutoSync definition.

NOTE
If Library Max Size was defined in the Library Selection Criteria
section of the Select Libraries for AutoSync screen, then the number of
libraries exceeding the defined limit will be displayed here.
8. If adjustments need to be made to the AutoSync definition:
a. Exit the results screen (F3).

b. Select F8=Selection
c. Select option 6=Reset Selection to reset the status back to “Defined”.
d. Make the adjustments by selecting option 2=Change on the definition.
e. After adjustments are complete, select option 8=Submit to resubmit
the definition so that the status indicates Complete.
9. Continue to create as many additional AutoSync definitions as is needed.
Remember to change the default journal as needed in order to balance the
journal load.
10. After creating all AutoSync definitions, verify they are in Complete status.
11. Press F12 to return to the main Library AutoSync screen (4.10). All libraries
included in any of the definitions created in the previous steps are
displayed with a Defined status.

12. Press F10 to set the status of all libraries to Ready (or select option 1 for
just a few libraries).
13. Select F11=SysSync Monitor to schedule the syncing of libraries. The

descriptions for each field appear in the table below the screen. Review and
make any necessary changes.
Field Instruction/Description
If replicating to multiple nodes, press F4 to select the

Target System
target.
The monitor job initiates the syncing process based on

Monitor Job Name the library definitions previously set up and settings in
the Scheduling section.
After all settings have been adjusted as desired, press

Monitor Status F18 to activate the monitor. The status in this field will
indicate Active.
Indicates how often the monitor job checks for libraries

Monitor Interval
eligible to sync.
Maximum Syncing Indicates the maximum number of active sync jobs

Jobs allowed to run on the node at one time.
Ready or Queued The number of libraries ready for syncing.

Requests
Warning Syncing Jobs Jobs in this field may be in a MSGW or HELD status.
A Y in this field allows the monitor jobs to run only on

Scheduling Activated
the specified days and during the specified times.

Field Instruction/Description
When the monitor is activated, it will only run on the

Active Days days specified with a Y. Enter an N on a specific day if
the sync jobs are not to run on that day.
When the monitor is activated, the sync jobs will run

only during the specified times. All zeros in the time
slots indicate that the sync jobs may run any time of
the day. Sync jobs that are currently running beyond
Active Times the designated time will finish running but no addi-
tional jobs will be started. The system will not allow
the time selection to span midnight. For example, you
can't start a selection at 11:00pm and have it end at
1:00am.
14. The System Syncing Monitor example below shows a reduction in the
number of syncing jobs on the primary and the scheduling activated for
weekdays between 1:00 am and 5:00 am.
15. When ready, press F18=Activate Monitor.
16. Once the System Sync Monitor is Active, press F12 to return to the
previous screen.
Depending on the scheduling parameters specified in the System Sync

Monitor, any libraries that are eligible to be replicated immediately will
begin to process.
The syncing status will indicate what process is being performed.
Press F5 periodically to watch the status of the libraries change as they are
synced to the target node. The list of statuses and their meanings is located
in the iTERA HA v6.0 Reference Guide.

Once the libraries have been processed and applied to the target node, the
library will be removed from the screen.
By default, only new libraries (i.e., libraries listed in the “New Libraries” view
in the 4.11 screen) are selected in AutoSync definitions. To build an AutoSync
definition for libraries not defined as “new”, do the following:
2. Position to E2LIBATOT.
3. Select option 2=Change.

4. Enter *YES for the parameter “Include Old libraries in Auto”, then press
Enter.

Non-Library Replication
5
Since not all users will replicate all non-library replication items, upon first
entering the Non-Library Replication menu, most menu options will not be
visible. They must be activated from the Replication Options menu.
Enabling Replication Options (30.23)

Access the Replication Options screen in the Environment and Setup Menu
(30.23) on the primary node and use option 6=Enable Global for each
feature you want to replicate. The Global State should indicate “On” for
each enabled option.
The replication options listed below are discussed in this chapter:
Option Description
*DEV Device Configuration
*DIRE Directory Entries
*IFS Integrated File System
*JOBSCDE IBM Job Scheduler
*MQ MQSeries/WebsphereMQ
*SPLF Spool File and *OUTQ
User Profile
NOTE
*USRPRF
The Global State for User Profile Replication
should already be on.
If the CRG contains more than one target node and you want to restrict
replication for a component to one of those nodes, use option 14=Disable
Local on that node.

Chapter 5: Non-Library Replication
The screen below shows several non-library replication options enabled (the
Global State displays “On”).
NOTE
When the Global State is toggled “On” on one node, it will be
automatically toggled “On” on the other defined nodes.
Option 8=Start Replication

Option 8=Start Replication in the Replication Options screen starts the
relevant jobs and should be selected for each of the following options that will
be replicated:
• Configuration and Devices
• Directory Entries
• IBM Job Scheduler
• Spool File and *OUTQ
• User Profile

Non-Library Replication Menu
Non-Library Replication Menu

With all Non-Library Replication options enabled, the Non-Library
Replication menu will be displayed as follows:
Instructions for general setup for each feature in the Non-Library Replication
menu are described in the next several sections.
User Profile (5.1)

Because User Profiles are critical for replication of libraries, instructions for
replicating this component were given in Chapter 4. See “User Profile
Replication (5.1)” on page 51 for details.

Directory Entries Replication (5.7)

IMPORTANT
Directory Entries must be replicated before IFS so that QDLS object
ownership information can be replicated to the target. If the
Directory Entry for the object’s owner is not replicated first, the
QDLS object owner information is not replicated.
This option allows you to define the replication of system Directory Entries
between nodes.
There are two types of entries with iTERA HA Directory Entry Replication:
• IBM i Directory Entries – These are created on the IBM i (ADDDIRE).
• iTERA HA Directory Entries – These are stored by iTERA HA for

replication and displayed when selecting menu option 5.7 (Directory
Entries Replication).
IBM only allows one directory entry to be created per user profile.
The user profile must exist on the node where the IBM i Directory Entry is
being created. If the user profile for the entry being replicated does not exist on
the target node(s) the iTERA HA entry will have an error status when an
attempt is made to replicate it.
NOTE
This program takes advantage of the advanced, feature-rich
capabilities of System Architecture. There are several additional
option and function keys available but not visible on the screens. To
display them, select menu option 50.2 (or execute the command
E2EXPLVL), set the experience level to 4, sign off, then sign back
on. The recommended level for general use is 2. These features are
documented in the appendices of the iTERA HA Reference Guide.

Automatic Replication of Directory Entries

1. If not already done, access the Replication Options menu (30.23) and
toggle the Global State for Directory Entries to On, using option 6=Enable
Global.
2. Select option 8=Start Replication, to start the xx_DIRREP job, which is

associated with Directory Entry Replication.
3. From the primary node, access the Directory Entries Replication screen
(5.7). The indicator “Exit Point Active” at the top of the screen will display
“No”. (When the Exit Program is not active it means that Directory
Entries are not being replicated.)
NOTE
Disregard the message “Data area E2DICTL in *LIBL not found”.
The data area will be automatically created.
4. Select F20=Control. The screen text indicates that only one CRG can
replicate Directory Entries. Specify the other parameters, then press F7 to
add the Exit Program.

NOTE
Disregard the message “Data area E2DIRPRD in *LIBL not found”.
The data area will be automatically created.
NOTE
Refer to the iTERA HA v6.0 Reference Guide for information on
the fields displayed in this screen.
5. When the Exit Program has been successfully loaded, “YES” will be
displayed in the Exit Point Active field. Press F3 to exit. (The exit program
is automatically enabled on the target node.)
6. Directory entry information from the IBM i must now be copied to a file
in iTERA HA. On the primary node, from the main Directory Entry
screen (5.7) press F18=Submit Info Build to retrieve the local IBM i
directory entry information. Press F5 periodically to refresh the screen
until the entries from the primary node are displayed.

NOTE
Only the primary node’s Directory Entries are displayed on this
screen.
7. When the entries have finished loading, select F16=Dft Map to display the
Directory Entry Mapping screen. The maps control how adds, deletes,
changes, etc. are handled on the target node.
8. The settings for the Q* map prohibit all directory entries beginning with
the letter Q from being replicated to the target node. (IBM directory
entries start with the letter Q should not be replicated.) Separate maps
must be created to replicate any profiles beginning with Q. If there are user
directory entries with names that start with Q, create a map for those
entries. Press F6=Add to create a new map, enter the information for the
directory entry, then press Enter.

NOTE
Adjustments may need to be made to replicated Directory Entries on
the target node in order for the entries to be viable after a role swap is
performed. For example, some third party applications require the
directory entry address to match the system name. Additionally,
SMTP routes may differ from primary to target. If either of these is
the case with your system, you’ll need to create a conversion map to
handle these issues. Follow the instructions in the section Conversion
Utility for Directory Entries. If this is not the case for your system,
continue with the next step.
9. To sync Directory Entries, do one of the following:
a. To Quick Sync all entries from the primary to the target system, on the
primary node from within 5.7, F16=Dft Map, select F21=Quick Sync
All.
NOTE
Any changes made to Directory Entries (for example, through
WRKDIRE) will be replicated to the target node.
b. To Quick Sync only select entries, on the primary node, access the
Directory Entries screen (5.7) and verify that the entries you want to
sync appear on the list, select option 21 (Quick Sync) for each entry,
then press Enter.
NOTE
Changes made to individually selected directory entries on the
primary will be replicated to the target node.
Conversion Utility for Directory Entries

The conversion map example below provides specific instructions for changing
the system name to the address field. However, there are many other uses for
the Convert Entry function. A common one is when the SMTP Route is
different between nodes. A conversion entry can be defined which will change
the route for the replicated directory entries. For additional information,
consult the iTERA HA v6.0 Reference Guide.
The sample entry below will cause the address field in all replicated directory
entries to change to the name of the target node.

To create a conversion map, do the following:
1. Select F19=Cvt Map from the main Directory Entries Replication screen.
NOTE
Use the F7=Load Default Map key to automatically create the
commonly used conversions for most systems.
2. Select F6=Add to create a new entry.
Field Description
Field The “field” that is being selected for conversion.
The value that will be converted (corresponding to

From Data
the “Field” field).
The new value that the “Field” field will be converted

To Data
to.
3. Select F4 to display the list of available fields. Select option 1 for the
Address field. (The screen automatically returns to the Directory Entry
Conversion display.)

4. Change the settings as indicated. Press Enter when finished.
a. Clear the From Node and To Node data.

b. Enter the name of the primary node in the From Data field.
c. Enter the name of the target node in the To Data field.
d. Enter Y in the Copy Both Directions field.
e. Press Enter.
5. When Enter is pressed, the entry will display as follows:
NOTE
Additional information on this screen is located in the Reference
Guide.

IFS Replication (5.2)

iTERA HA uses journaling to replicate most Integrated File System objects.
However, object-level replication is used for objects that cannot be journaled
(such as QDLS).
NOTE
Before you can access IFS Replication screens, you must enable IFS
replication in Replication Options (30.23 opt 6) under the
Environment & Setup menu.
NOTE
Directory Entries must be replicated before IFS so that QDLS object
ownership information can be replicated to the target. If the
Directory Entry for the object’s owner is not replicated first, the
QDLS object owner information is not replicated.
IMPORTANT
IBM documentation for V5R3 of i5/OS states “The maximum
number of objects that can be associated with one journal is 250,000.
This maximum includes objects whose changes are currently being
journaled, objects for which journaling was ended while the [journal
receivers that are still in the current chain with the currently attached
receiver]. If the number of objects is larger than this maximum,
journaling does not start”.
If you attempt to journal a directory with more than 250,000

objects, the objects will not get created in the IFS and your
applications may fail.
Before assigning directories to journals and starting journaling, you

must research your directory structure to determine the best way to
journal the IFS, determine the current number of objects in a
directory as well as anticipate the number of new objects that may get
added to a directory.
If you have a directory that has more than 250,000 objects in it you
can either mirror it using object-level replication or journal
subdirectories to separate journals.
The maximum number of objects that can be associated with a given

journal for V5R4 of i5/OS is 10,000,000.

IMPORTANT
If in the event that syncing is cancelled for a directory on the
primary, journaling should be ended on the equivalent directory on
the target (otherwise, there is the potential that the number of
allowable objects in the journal could be exceeded). If the objects
that had previously been mirrored should not reside on the target,
they should be manually deleted.
Important Information About Replicating IFS

Before initiating replication of IFS, you must fully understand the directory
and object organization within the IFS and know what you want to replicate.
All IFS to be replicated must be assigned to a default journal (including IFS
that will be replicated using object-level replication).
Objects in the root directory may be replicated using either object-level or

journaled replication. However, objects in QDLS may only be replicated using
the object-level replication.
Object-level replication sends an entire object to the target system if any data
in the file changes.
NOTE
Certain directories and objects within IFS should not be replicated.
Most of these directories are not displayed in the IFS Replication
screens. Consult the iTERA HA v6.0 Reference Guide for a partial
list of IFS object types and directories ineligible for replication.
Additionally, directories that have licensing information in them

should not be replicated. (If licensing information were to be
mirrored, the target node’s licensing would be lost (replaced). For this
reason, they are not displayed in the All Available Directories view.)
Journaling or Object-level Replication – How to

Choose
Several factors will influence whether to use journaling or object-level to
replicate IFS. One of the most important things to remember about journaling
is the maximum number of objects that can be sent through the journal. For
V5R3 of i5/OS, that number is 250,000. If more than 250,000 objects are
journaled, problems arise, as described in the previous warning. Naturally, if
there are directories well under that number but to which objects are
constantly being added or deleted, the maximum number of objects that will
be assigned to a given journal must be considered. V5R4 of i5/OS increases the
limit to 10,000,000, so the issue with journal object capacity is not typically an

issue. The actual limit is what is specified in the Journal Object Limit
parameter of the CHGJRN command.
The inherent dilemma in replicating IFS is in determining the volume for a

directory which will be replicated. There is no clear way to determine the
volume for IFS objects being journaled until they are actually journaled. Once
journaling has begun, the journals can be reviewed in order to determine
volume. Also, ending journaling for heavily used objects may not be possible.
However, on the other hand, when using object-level replication, each time an
object changes, the entire object is sent to the target node. Thus, there may be
a higher volume going through the journal than is desired and higher volume
may impact system performance. Some objects rarely or never change, so using
object-level replication may be more optimal. For these reasons, it is important
to have a good understanding of the IFS prior to replication.
Replication Mapping Strategy

One way to learn more about the characteristics of the IFS is to create what
we’ll call a replication mapping strategy. In a replication mapping strategy, the
directories that have been chosen for replication and the level of the directory
structure being replicated is kept track of by collecting screen shots or copying
text into a document. In developing a replication strategy, there are two
methods to consider.
• Option 19 from within IFS replication (e.g., 5.2, opt 5, opt 5 on root, opt
19) is used to display the total file size, number of directories, number of
subdirectories, and number of symbolic links contained in the file.
• The display file command for the root directory may also be used to access
a list of all IFS on the system (DSPF ‘/’).
A benefit of using option 19 is working within IFS Replication to view the

directories. Also, several of the directories ineligible for replication are
pre-filtered and do not display in the list. The output is displayed on one line
at the bottom of the screen several directories may be selected at one time. To
navigate quickly through IFS directories, after selecting option 19 for several
directories, press Enter, then press Page Down to advance to the next directory
output.
The benefit of viewing the output of the DSPF / command is that several
directory results can be viewed at one time, making it easier to identify groups
of smaller directories and to isolate the large directories in order to balance the
journal load.
Regardless of the method used, the goal is to control the amount of data going
through the journal. If there are too many directories or excessively large
objects going through one journal, the apply job may fall behind.

Since every system is different, there is no standard recommendation on

journal loading, so redistribution of directories in the journals may be needed.
Option 19 Output
Within IFS Replication, select option 5 to view a file system. Select option 19
for various directories (multi-select is supported). Drill down using option 5 as
needed within subfolders to determine which level of the directory structure to
replicate.
The output for option 19 displays at the bottom of the screen as such:
Size: 35,852 Nbr Dir: 1 Nbr Files: 1 Nbr SymLnk: 0
With the information about directory path size, the number of subdirectories,
files, and symbolic links, make a determination of how many directories will be
replicated at one time.
IMPORTANT
This selection process is wholly dependent upon the particular
system and bandwidth. If large directories are selected (or too many
directories at one time) to replicate, a critical bottleneck will occur.
DSPF ‘/’ Output:

Position the cursor on one of the directory names and press F16 to arrange the
directories in alphabetical order.
For path size information on each directory to be replicated, enter option 6. In

the resulting display, the Size column displays the total path size. The CCSID
or Symbolic Link column displays the total number of subdirectories and the
total number of files (and objects).
Do not replicate system-loaded directories, such as those owned by QSYS.

However, drill down into the subfolders to determine if there are folders that
do need to be replicated.
The total number of subdirectories and files/objects does not include any
symbolic links within those directories.

• Look for large directories by checking the total path size, the number of
subdirectories, and the number of files/objects. (Large is relative to your
system.)
• Do screen prints or screen captures of the various screens and note the
intended syncing strategy.
What to Replicate
When IFS is replicated in iTERA HA, any objects located within a replicated
directory, including all subdirectories and objects within those subdirectories,
will be replicated. If a higher-level directory containing, for example,
subdirectories that contain licensing information is replicated, there will be
problems. Drill down into each subdirectory in order to determine if all items
within the directory are eligible for replication.
The following file types may be replicated using IFS:
*DOC Document
*FLR Folder
*DIR Directory
*STMF Stream File
*SYMLNK Symbolic Links
IFS Object Types Ineligible for Replication

The iTERA HA v6.0 Reference Guide contains the list of directories ineligible
for replication.
Replication Instructions
Prior to actual replication, one or more IFS journals must be created, the
journaling defaults for the IFS journals must be adjusted, and the first journal
to which IFS objects will be replicated must be designated as the default
journal. These steps are common to both journaling and object-level
replication. Once these steps are completed, continue with either the
“Replicate IFS Using Journaling” on page 115 or “Replicate IFS Using
Object-Level Replication” on page 116.
Create an IFS Journal

Create one or more IFS journals from within the 3.1, Work with Local
Journals screen on the primary. As mentioned earlier, it is difficult to determine
how many journals will be needed for IFS Replication until the objects are
actively being journaled. For now, simply estimate how many will be needed.

1. From 3.1 (Work with Local Journals) on the primary, select F6=Create
Mirror Journal.
2. Specify I for the New Journal Type field, then press Enter.
IMPORTANT
The New Journal Type must be “I” for IFS journals.
3. Type in the Description, press Enter.
Naming conventions for IFS Journals:

• The first two characters are the two-letter CRG code.

• Third character is an I for IFS Journal.
• Fourth and fifth characters are JR, for journal.
• Sixth character differentiates the IFS journals from each other and is
increased by one character for each additional journal.
4. Select option 32=Toggle Mirror Status to “On”.
5. On the target node, select 3.4, Apply Job Maintenance. The apply job for
the journal just created will be displayed with the Jrn State “*ON” and the
Process State “OFF”.
6. Select option 11=Suspend Process State.
7. Override the next sequence number to process by selecting option

12=Override Next Seq. Press F7 to continue.

8. Since this is a new Journal, select option 3, “Use the first seq# of the
current attached receiver” then press Enter.
9. When prompted whether to delete prior receivers, answer “Y”. Type the
reason for change, then press Enter.

10. Press F5 until the Process State changes to Active.
11. Select option 14=Restart Job to start the apply job. Press F5 periodically. If
the apply job has started correctly, the job status should change to either a
RUN status or an EVTW status.
12. Verify that remote journaling was added. On the primary, (still in 3.1)
select option 5=WRKJRNA for the new journal, then select F16=Work
with remote journal information. The Journal State should be *ACTIVE.

Journaling Defaults, Replication Options, and Assigning a

Default Journal
Now that at least one IFS journal has been created (create more as needed), this
next section includes instructions for adjusting the defaults for for the journal,
reviewing the replication options, then designating the first journal (the default
journal) to which IFS objects will be replicated.
1. On the primary, select 5.2. The Work with File Systems screen is
displayed.
NOTE
If the QDLS directory cannot be accessed, it is due to an authority
issue with the User Profile that is signed on. Add the User Profile to
the Directory Entry (WRKDIRE).
2. Press F7=Journaling Defaults. The defaults displayed are the defaults used
for IFS Replication and generally will not need to be changed. However,
review the Help text (F1), change as needed, and press Enter.
3. The IFS Replication Options screen is displayed. Review the Help text
(F1) change as needed, and press Enter to return to the main screen.

4. Select Option 5=Work with file system to drill down one of the file
systems.
5. The All Directories view is displayed. Press F13=Select Default Journal.
NOTE
To replicate objects using object-level replication, a default journal
must still be assigned to the object or directory. The assigned journal
is the journal through which the objects will be sent to the target
system.
Only IFS journals are displayed. Select one to be the default journal for the
chosen directory. Remember that all subdirectories will be replicated
(unless specified otherwise in IFS Journaling Defaults). Select option
1=Select with Default Image for the journal and press Enter.

6. The screen is automatically returned to the All Directories view and the
default journal is displayed.
7. Select option 5=Display link for the designated file system to drill down
further to view subdirectories and objects (refer to the replication mapping
strategy).
8. Select option 1=Assign to dft jrn for the directory you want to replicate.
The directory (and all objects within it) is then assigned to the default
journal.
9. Continue with either “Replicate IFS Using Journaling” on page 115 to

replicate directories/objects using journaling or“Replicate IFS Using
Object-Level Replication” on page 116 to use object-level replication.
• If using journaling, when replication is started, the directory and all its
objects will be replicated using the assigned journal.
• If using object-level replication, information about the object is sent to
the target node through the journal.
IMPORTANT
Some applications may get errors because they don’t have adequate
authority to the journal to which IFS objects are being journaled. To
avoid this, grant user *PUBLIC, *ALL authority to the journal. Do
this after the journal is created and before you start journaling.

Replicate IFS Using Journaling

IMPORTANT
250,000 is the maximum number of objects than can be associated
with the journal in V5R3 of i5/OS.
1. Select option 2=Start jrn to start journaling the directory. The Status will
indicate journaling is active.
NOTE
It may take a long time to start journaling on large objects.
2. After journaling has started, select option 3=Start Mirroring to start

mirroring. A confirmation window will be displayed if the confirmation
notification defaults were not modified. Answer Y, then press Enter. The
Status will indicate Active when the directory is actively being mirrored.
The text color will change to pink.
NOTE
As an alternative to selecting options 2 and 3 separately, option
4=Quick sync (2,3) may be used to start both journaling and
mirroring for each object to be replicated. Journaling will be fully
started before replication is initiated. Keep in mind that if there are a
large number of objects in the directory, it may take a long time to
start journaling.

Replicate IFS Using Object-Level Replication

NOTE
To replicate objects using object-level replication, the object or
directory must still be assigned to a journal (option 1). The assigned
journal is the journal through which the objects will be sent to the
target system. To use object-level replication, first complete the
applicable steps in the section “Replication Instructions” on
page 107, above. Once the journal has been assigned, continue with
the object-level replication instructions.
1. Enter option 14=Use object-level mirroring on the object to be replicated.

After pressing Enter, a Y will be displayed on the item in the OL column.
2. Start replicating the objects by entering option 3=Start Mirroring on the

object and pressing Enter. The Status column will indicate Active and the
text color will be pink.
NOTE
Options 2, 4 and 8 are not valid for object-level replication.
IFS Audit
The IFS audit reviews the data, attributes, and authorities. The audit will run
automatically via the Audit Command Console; however, to manually audit
selected files or directories, use option 20=Audit from the 5.2 screen or execute
the command E2IFSAUD on the primary. The directory to audit and the

parameters are displayed and may be adjusted, if needed. Press Enter to initiate
the audit.
View the IFS audit results in the Audit Command Console (1.8) on the target
node by selecting option 1 on IFS_REVIEW. The IFS Audit History screen
provides summary information about IFS audit history.
Select option 1 on the report to view. The Display IFS Audit History screen is
displayed.
Place 5 by an object on the next screen to view audit detail on the selected
object.

The report displays audit result by “Passed” and “Failed” statuses. The report
also shows difference and corrected values. See the section “Audit Detail” on
page 185 for additional information on viewing the IFS Audit results.
Purge Audit Data

Automate the IFS audit deletion process by scheduling the E2IFSPRGA
command to run weekly on the target node. However, if IFS is audited more
frequently, schedule it to run based on how often the IFS audit is run, e.g., if
the audit runs daily, schedule the purge to also run daily. Specify the number of
days of audit information you want to retain, as well as the number of days to
retain data on failed objects. The specified default of retention for 8 days is a
general recommendation and may be adjusted, based on personal preference.
Adjust the Scheduled Time as desired.
Ending IFS Replication

To end IFS Replication, select option 7=Cancel Mirroring, then option 8=End
Journaling, then option 9=Remove Jrn Assignment, or select option 10=Quick
End (7, 8, 9), which executes options 7, 8, and 9 consecutively.
NOTE
Ensure that synchronization is complete before ending mirroring on
a sub directory. An indication of complete synchronization is that the
record in the sub file will turn a different color (usually pink) and the
status will be active.

Out Queue & Spool File Replication (5.3)

This option is used to configure and execute the process of replicating output
queues and spool files from the primary node to the target node(s). When this
option is taken, the Output Queue Search screen is displayed, from which
individual output queues are replicated to the target node. Once defined,
designated output queues continue to be replicated automatically until they are
toggled off.
NOTE
Spool File Replication Instructions

1. If not already done, access the Replication Options menu on the primary
node (30.23) and toggle the Global State for Spool File Replication
(*SPLF) to On, using option 6=Enable Global.
2. Select option 8=Start Replication on the xx_RPTREP job.

3. Select Work with Subsystem Jobs (2.11 or E2SBS).
4. End all xx_OBJMON jobs using option 4 (where xx is the two-character

abbreviation of your CRG), press Enter to confirm. Refresh the screen and
verify the jobs have ended, then press F12 to exit the screen.
5. Restart the xx_OBJMON jobs by executing the command E2STRMON.
6. Display the subsystem (E2SBS) and press F5 until the job xx_RPTREP is
displayed.
7. On the target system, select Work with Subsystem Jobs (2.11 or E2SBS).
8. End all xx_OBJMON jobs using option 4, press Enter to confirm. Refresh
the screen and verify the job has ended, then press F12 to exit the screen.
9. Restart the xx_OBJMON jobs by executing the command E2STRMON.

10. In the iTERA HA subsystem (E2SBS), verify that the job xx_RPTREP is
running. If using i5/OS V5R3, also ensure that the xx_RPTCHG job is
running.

The Mirrored Output Queues view is the default display.
12. Press F18=Submit Analyzer. Press F7=Toggle Views until the All Output
Queues view is displayed. The list will be populated as the Analyzer runs.
13. Select one of the following options for each output queue, then press
Enter.
– 11=Repl (Old and New) Replicates the output queue, existing reports
and new reports as they are added.
– 12=Repl (New only) Replicates the output queue and new reports as
they are added. Existing spool files in the output queue are replicated
only when the audit is run.

When Enter is pressed the Status field will display Active, and the Rep
field will display Yes, indicating that replication is active.
14. Optional: select option 15=Remove New Status for any output queues you
do not wish to replicate. This will remove the “New” indicator from the
Status field.

Configuration Replication (5.4)

The Configuration Replication process allows the operator to designate if (and
how) individual configuration components (lines, controllers, devices, class of
service, network servers, and modes) are replicated to the target node. Selected
components continue to be replicated automatically until they are toggled off.
All components required for running a device being replicated on the primary
node are also automatically replicated to the target node. For example, if a
device which has a controller attached is replicated, the controller is also
automatically replicated. If a printer is attached, and the printer isn’t already
defined on the target node, it will replicate it automatically.
Lines aren’t automatically replicated because manual manipulation is usually

required in order to make replicated lines functional on the target node.
Therefore, it is usually best to make certain the lines are copied to the target
node then manually adjusted prior to selecting controllers and/or devices for
replication.
When this screen is first accessed, no devices will be displayed but the system
will automatically begin building the list. Select F18=Submit Info Build in
order to ensure the list is fully updated and displays all current devices.
NOTE

Replicating Configuration Components

node (30.23) and toggle the Global State for Configuration and Devices
(*DEV) to On, using option 6=Enable Global.
2. Select option 8=Start Replication, to start the xx_DEVREP job.
3. Enter the Configuration Replication screen (5.4) on the primary node.

The first time this screen is accessed, no devices will be displayed (but the
system will start populating the list). Build the list by selecting
F18=Submit Info Build.
4. After the list is built, the lines, class of service descriptions, network serves,
and modes are displayed in the Config Name column.
NOTE
The *BLANK category contains controllers that do not have
attached lines. Within the *BLANK category for lines is another
*BLANK category for devices that do not have attached controllers
(not displayed in this view).
5. Select F16=Expand/Compress to display all controllers and devices. To

expand the view for individual items, select option 11=Expand. This
displays the next level. (Option 11=Expand on a line displays the
controllers attached to the selected line. Option 11=Expand on a
controller displays the attached devices.)
– Lines, controllers, and devices are displayed in a hierarchical tree

structure.
– Controllers are indented slightly from the lines and devices are
indented further from the controllers.
– The plus sign (+) to the left of a line indicates there is at least one
controller attached to it.

– The plus sign (+) to the left of a controller indicates there is at least one
device attached to it.
6. Review the settings in the F20=Control screen. Typically, most settings in

this screen will not need to be adjusted.
For a description of each field, press F1 on the field or consult the

Reference Guide for additional information on this screen.
IMPORTANT
Replication can be done globally or individually. Read through the
instructions in the following two sections before determining the
best way to proceed. Complete one or both of the following sections.
Creating Maps to Globally Replicate Groups of

Configuration Components
1. Press F8=Default Config Map to display the Configuration Defaults
Maintenance screen.
Configuration Defaults Maintenance controls how adds, deletes, and

changes to controllers and devices are handled and allows devices to be
replicated based on name versus by controller.

A typical setting for the *ALL map is to set adds and changes to Y but to
leave deletes set to N. (The default *ALL map is set to NOT update the
target configuration components when adds, changes, and deletes occur—
this is a precaution in order to prevent someone from unintentionally
sending everything across without considering the ramifications.)
You must determine if you wish to set up a more specific map to replicate
certain controllers and/or devices, or you may skip this step and go back to
the main Configuration Replication screen, where you can select
individual entries to replicate.
In the example below, a map covering devices starting with “BI” (for
billing) and only those with the category of *DSP was created for the
billing department.
This map is defined to update the target node every time adds, deletes, and
changes are detected on the primary node.
2. If setting up an additional map on your system, press F6=Add. The

following screen is displayed. Fill in the Configuration Default
Maintenance screen with appropriate data, then press Enter.
NOTE
A list of Categories is available in the Reference Guide.
3. With the appropriate maps in place, from the Configuration Defaults

screen (5.4 F8), press F21=Quick Sync All to sync all entries covered by
the defined maps to the target node.

4. After the desired maps are adjusted and/or created, use F8=Copy Maps to
sync all defined map entries to the target node.
5. Press F12 to return to the main Configuration Replication screen.
NOTE
The Sts column displays the status of replication. If not all Lines,
Devices, Controllers, Class of Service, Modes, and Network
Descriptions that should be replicated to the target node were
covered by the maps, continue with the next section.
Selecting Individual Configuration Components

for Replication
The key elements in replicating individual configuration components is to
understand how the settings in the Rep (Replicate Change) and Ovr (Allow
Device Override) columns affect replication.
The color coded data, along with the Sts (Status) column, will help to clearly
see what has been replicated and what has not. The color codes are as follows:
Color Description
Green Replication has not yet been started for an object for the first time.
Pink The object is actively being replicated.
Red Replication has been manually stopped for an object.
The Ovr (Allow Device Override) column specifies whether a particular device
is eligible to be replicated.
When set to Y, the object is designated as eligible for replication.
• When set to N, the object is not eligible for replication. This value is
typically used in only two circumstances: 1) if you want to ensure that
replication is not accidentally started for an object (that is not currently
being replicated)—even if option 21=Quick Sync is attempted; and 2) if
you want to stop or prevent replication for a device that is associated with
a controller when a Y appears in the Allow Device Override column.
When initially starting replication for an object, type a Y in the Rep column.
Once this screen is exited, the Object Monitor in iTERA HA will recognize
this Y in its next cycle of object monitoring and will then begin replication of
the objects. If, however, you want the object to be replicated immediately, use
option 21=Quick Sync in the adjacent option field and press Enter.

All components required for running a device being replicated on the primary
node are also automatically replicated to the target node. For example, if a
device with an attached controller is replicated, the controller is also
automatically replicated. If a printer is attached, and the printer isn’t already
defined on the target node, it will replicate it automatically as well.
• If the configuration is a controller and you want to individually choose

whether the devices associated with the controller are to be replicated, type
a Y in the Allow Device Override column. This will automatically mark all
associated devices as ineligible for replication (N is displayed in the field).
However, the field next to each device will become input capable, allowing
you to choose which devices to replicate using option Y.
• If the configuration is a controller and you do not want to individually

choose whether the devices associated with the controller are to be
replicated, type an N in the Allow Device Override column. This will
automatically mark all associated devices as eligible for replication (Y is
displayed in the field), and this field next to each device will become input
capable.
When you specify Y in the Replicate Change column for an object, it will be
replicated as soon as the Object Monitor detects it (which may take up to 15
minutes).
Two options for syncing immediately:
– Option 21=Quick Sync will replicate selected objects immediately

(press Enter).
– Option 22=Quick Sync w/devices, if done at the controller level, will
replicate all devices for that controller. The controller’s status will
change to Active and the text will change to pink. This can also be
done at the device level.
Ending Replication for Configuration Components

There are two ways to stop replication for Configuration Components:
• Select option 14=Stop Mirror. This will only stop replication of the
selected line, controller, or device. Those devices will remain on the target
node but will not be monitored for updates.
• Select option 4=Stop Mirror and Dlt Rmt Cfg Objects. This option will
not only stop replication of the selected line, controller, or device but it
will also delete the controller and/or devices on the target.

Job Scheduler Replication (5.5)

Job Scheduler Replication is designed to ensure that you have the same entries
in the Job Scheduler on all nodes.
IMPORTANT
iTERA HA v6.0 only supports Job Scheduler Replication from the
primary to the target system. Bi-directional replication is not
supported in this release. However, entries can be manually copied
from the target to the primary using the Remote Job Schedule
Maintenance screen.
One of the convenient tools within Job Scheduler Replication is the ability to
link a job on the primary with the same job on the target node (the job is
placed in Held status on the target node). This should be done prior to
replicating jobs from the primary to the target in order to prevent sending
duplicate jobs to the target node and for easy identification of jobs.
In Job Scheduler Replication, jobs can be copied both to and from the primary
and target nodes, the job schedule entry can be updated, and more.
A convenient way to create a scheduled job that runs on the target is to first
create it on the primary then replicate it over. This allows you to manage the
scheduled job from the primary and have it replicated to the target. See
“Manage Scheduled Target Jobs From the Primary” on page 133 for
additional information.

NOTE
How to Set Up Job Scheduler Replication

node (30.23) and toggle the Global State for IBM Job Scheduler
Replication (*JOBSCDE) to On, using option 6=Enable Global.
2. Select option 8=Start Replication, to start the xx_JBSREP job, which is

associated with Job Scheduler Replication.
3. Enter menu option 5.5 on all nodes the first time Job Scheduler
Replication is started in order to enable iTERA HA to automatically adjust
an audit level (to *CHANGE). This lets QUADJRN know about changes
to the job scheduler.
4. On the primary, select F16=Map to display the Job Schedule Mapping
Search screen. The Job Schedule Mapping Search screen controls how jobs
are handled.
The default map entries in this screen are as follows:
Map Entry Description
This map handles all other jobs not covered by other

*ALL
maps.
Controls the scheduling of the E2CLRSNCOB

E2CLRSNCOB
command on the target.

Map Entry Description
“Q” jobs are generated by the system and should not

be replicated. The values for Add, Delete, Change for
Q*
the Q* map are all set to No and the Default Status is
“Held”.
Controls the execution of the user activation

QSECACT* schedule. The Add Job field is set to H, which hides it
from view within the job scheduler.
Controls the execution of user expiration. The Add

QSECEXP* Job field is set to H, which hides it from view within
the job scheduler.
The journal manager job, which runs at different

times on the primary and target, should not be
XPJRNMGT
replicated. The Add Job field is set to H, which hides
it from view within the job scheduler.
When option 1 is selected for a map, the Job Schedule Map Maintenance
screen is displayed (see below).
Item Description
Add Job • Y - The jobs covered by the map entry will be

added to the target node.
• N - The jobs covered by the map entry will not be
added to the target node.
• H - Hides the job from view on the list.
Change Job • If set to Y, jobs that are changed on the primary

will be changed on the target.
Delete Job • If set to Y, jobs deleted on the primary will be

deleted on the target.

Item Description
Default Status • SCD - Indicates that if the job is scheduled on the

primary, it will be scheduled on the target (and
vice versa).
• JOB - The status will reflect the status the job was
in on the original node.
• HLD - The copied job will be placed in held
status, regardless of the status of the job on the
original node.
Log Changes When set to Y, logs an entry for each change made to
a job and every time a job is run. The usual setting is
N.
5. Press Enter to accept any changes then F12 to exit. If changes to the map
were made, do the following:
a. End the xx_JBSREP job in the subsystem.

b. Select 2.20 to restart the Job Monitor.
c. For changes to jobs that already exist, remove the job from the target
(5.5, F10=More Functions, option 10=Remote Job Scd, option
9=Remove From Backup) then initiate an audit by selecting 5.5, option
22=Audit on the primary. The job will now use the new map.
6. From the Job Schedule Entry screen, select F18=Submit Info Build. This
will load the list of jobs on the primary (local) machine and identify them
to iTERA HA.
7. Press F10=More Functions, select option 10 Remote Job Scd, then press
F10=Retrieve Remote to retrieve the list of jobs in the job scheduler on the
target node to the primary node. Select F5 to refresh the screen.
8. If the jobs on the target node are the same as those on the primary (either
manually created or already existing), then you have the choice to either
link the jobs as they are, or remove the jobs from the target system.
IMPORTANT
Failure to do this step correctly will result in duplicate jobs being
created on the target.
a. To link the jobs on primary to the equivalent job on the target, from
the main Job Scheduled Entries screen (5.5) select F16=Map to view
the Job Schedule Mapping Search screen, then select F20=Link All.
This will submit a job that will link jobs on the primary to those on the
target.
b. To remove the jobs from the target: either sign on to the target, execute
WRKJOBSCDE, then delete them, or, on the primary, select
F10=More Functions, opt 10 Remote Job Scd, then select option
9=Remove from Backup for all jobs to be removed (to remove them all,

select option 9 for the first entry, then select F13 to repeat the option
for all subsequent jobs in the list).
9. Select F20=Link All from the Job Schedule Mapping Search screen (5.5
F16) to compare jobs between the primary and target and link
corresponding jobs. This will prevent duplicate jobs from being copied to
the target node when the quick sync is performed. Press F12 to return to
the Job Schedule Entry screen.
10. Select either F21=Quick Sync from the Job Schedule Mapping Search
screen (5.5, F16, F21) to replicate all jobs to the target node, or, to
individually select jobs to sync, select option 21=Quick Sync from the Job
Scheduled Entries screen (5.5, opt 21).
When jobs have been synced, they will be color-coded to indicate the
status of replication:
• RED – An error exists. Usually indicates the job doesn’t exist on that
node.
• GREEN – The Job Schedule Entry has no match on the other node.
• BLUE – Job Schedule Entry is actively being replicated.
Other Job Scheduler Replication Features

Convenient access for updating scheduled jobs is provided so that if changes to
a job are needed, they can be accomplished quickly and easily. Simply select
option 1=Update for a job and it will display the following screen:
If you need to work within the actual Job Scheduler, select option
5=WRKJOBSCDE for a job or select F8=WRKJOBSCDE.
If you need to delete an entry from the Job Scheduler, do so from within the
Job Scheduler (option 5=WRKJOBSCDE), then option 4=Remove).

NOTE
Option 4=Delete from list used from the main iTERA HA Job
Scheduled Entries screen will only remove a job from being defined
in iTERA HA but will not delete the actual job from the Job
Scheduler.
Another convenient tool is option 3=Copy Jobs, which allows you to create a
new job on the same node by copying an existing job. Simply select option 3
for the job you wish to copy. The following screen is displayed.
Type in the name of the job and press Enter.
Manage Scheduled Target Jobs From the Primary

A convenient way to create a scheduled job that runs on the target is to first
create it on the primary then replicate it over. This allows you to manage the
scheduled job from the primary and have it replicated to the target. Refer to
the instructions at the end of the Job Scheduler Replication section for
additional information.
1. Create a map on the primary (5.5, F16=Map). Set the map as follows: Add
job=Y, Change job=Y, Delete job=Y, Default status=SCD.
2. In the iTERA HA subsystem (E2SBS) on primary, end the xx_JBSREP

job.
3. Start the job monitor using menu option 2.20.
4. From 5.5 on the primary, select F10=More Functions, then option

10=Remote Job Scd.
5. Copy the jobs using option 3=Copy Jobs. Leave the name the same as the
original name, this will copy it to the primary system. Press Enter, then
F12 to exit the screen. The job should now be visible in the 5.5 screen in
syncing status.
6. Select option 5=WRKJOBSCDE.

7. Put the job on hold using option 3. This will put the job on the primary
on hold so that it does not run. Return using F12.
8. Select option 31=Toggle Prim Status.
9. Select F10=More Functions, then option 10=Remote Job Scd.

10. Press F10=Retrieve Remote to retrieve the remote jobs.
11. Press F5 to refresh the screen. The job will now be listed twice on the
target.
12. Select option 9=Remove from backup on the job that is not linked. Use
F12 to return to the Job Scheduled Entries screen.
13. Select option 22=Audit to audit the job.
14. Select option 23=Link Jobs to link the job again.
Create a New Job on the Primary

1. Create a map on the primary (5.5, F16=Map). Set the map as follows: Add
job=Y, Change job=Y, Delete job=Y, Default status=SCD.
2. In the iTERA HA subsystem (E2SBS) on primary, end the xx_JBSREP

job.
3. Create the job on the primary and put in on hold.
4. Select option 31=Toggle Prim Status.
5. Select F10=More Functions, then option 10=Remote Job Scd.

6. Press F10=Retrieve Remote to retrieve the remote jobs.
7. Press F5 to refresh the screen. The job will now be listed twice on the
target.
Job Schedule Replication Errors (5.5 Target)

Job Schedule errors can be viewed only on the target system. The count
indicates the number of times the error was found. Job Schedule errors are
usually caused by a missing or incorrect job description, job command, or user
profile.

Duplicate jobs in the Job Scheduler Replication

Screen (5.5)
Duplicate jobs can occur in the Job Scheduler Replication screen for a number
of reasons:
• Issues occur during iTERA HA upgrade.
• User error in replication.
The command E2PRGJSDUP can be used to remove these duplicate job

scheduler entries from the primary.
1. Enter the command E2PRGJSDUP on the primary system, press F4 to

prompt the command. The following screen is displayed:
2. Entering *YES for the Prompt Command parameter and pressing Enter will
display each duplicated job schedule entry on the Remove Job Schedule
Entry Screen, as follows:
3. Press Enter to process each duplicated entry.
4. If *NO is used for the Prompt Command parameter then the duplicate
entries are automatically removed; the RMVJOBSCDE screen will not be
displayed.

Ending Job Scheduler Replication

To end Job Scheduler Replication, toggle the Global State off in Replication
Options (30.23).
Role Swap Considerations for Replicated Jobs

The exit programs E2USRDWN and E2USRUP are used to execute programs
or commands during the role swap. The program E21440RP1 sets the status of
job scheduler entries on both the new backup and new primary. They have
been added to both exit programs, but commented out. You will need to
change the programs if you want to change the status during a role swap.
Other Scheduled Items That Can Be Replicated

The iTERA HA job scheduler replication option supports not only scheduled
jobs but also User Activation schedules, User Expiration schedules, and Power
On/Off schedules.
These other types of jobs can be accessed by pressing F10=More Functions

from the main Job Schedule Entries screen, then selecting the appropriate
option from the window that is displayed. These are IBM features. Refer to
IBM documentation for further information on these features.

WebSphere MQ Replication (5.6)

iTERA HA supports replication for MQ v6.0 and WebSphere MQ v5.3.
WebSphere MQ Overview
WebSphere MQ is a communication system that provides assured
asynchronous, once-only delivery of data across a broad range of hardware and
software platforms. It is used for data transfer from one system to another for
interoperability of sharing data.
WebSphere MQ has three basic components:
• Messages – The data that is sent from one application to another.

• Queues – The data structure where messages are stored.
• Queue Manager – Provides queuing services to applications and manages

queues assigned to it.
Checkpoints are created to establish a point of recovery. They are created at the
point when everything in a queue has been stored in a file.
WebSphere MQ stores information in both libraries and in the IFS:
• Libraries contain a program, a file, two message queues, a journal, and

journal receivers for each queue manager.
• The IFS contains configuration information and stream files (which

contain data).
WebSphere MQ extensively uses journals. Every message is sent to the journal

before it is sent to the queue. All journals used by MQSeries are created by
MQSeries.
In MQ journaling:
• Objects are not journaled to the MQ journal.
• All journal entries are user entries created by MQ.
• iTERA does not put any journal entries in the MQ journals.
• All journals have the same name—AMQAJRN.
• Journal entries are the most critical part of MQ in a failover or role swap.
WebSphere MQ Replication
All WebSphere MQ replication is performed on the primary node.
IFS and library replication (pertaining to MQ) is initiated by the MQ

replication feature.

One ‘Controlling’ job runs on the target node (ZM_mqdftjrn) and one apply
job runs for each MQ Queue Manager (ZU_AMQAJRx). There are no
additional jobs run on the primary node.
IMPORTANT
Verify that the user profiles QMQM and QMQADMIN exist on the
target node prior to initiating replication. If these profiles do not
exist and MQ is replicated, all authorities are lost.
How iTERA HA keeps WebSphere MQ replicated in real time:
1. Replicates the associated library.
2. Creates the Queue Manger, if needed.
3. Creates the queue.
4. Replicates the current messages and other IFS objects.
5. Creates the remote journal and receiver.
6. Applies the messages to the target node, then deletes messages as needed.
WebSphere MQ Replication Instructions

1. The Global State must be toggled on for both IFS and WebSphere MQ
(30.23, option 6=Enable Global).
2. Select menu option 5.6 to access WebSphere MQ replication options. The

Change Queue Manager Journal screen will be displayed. Use option 2 to
select a Queue Manager.
NOTE
A “Y” in the Jrn Active column indicates journaling is active.

3. Select F13 to assign the default journal. The New MQ Default Journal
screen is displayed. Select F6 to create a new MQ Journal.
4. Specify M for the Journal Type; press Enter.
5. The following screen is displayed. Enter a description for the Default

Journal, then press Enter.
6. Select option 1 on the journal you just created. The Default Journal is now
defined. Select option 1 for each Queue Manager to assign the Default
Journal to the Queue Managers.

7. When Enter is pressed, the main MQ Replication screen is displayed:
8. Press F14 to start journaling the IFS. Note the status of journaling is
displayed in the Status field.
When Journaling is started:
• The status changes.

• The Jrn Active column displays Y.
• Options are disabled.
• A completion message is displayed.

9. Press F15 to start replication. Status messages will display during start up.
F15 starts replication of MQ IFS directories and Queue Manager libraries.
10. The status of journaling is highlighted in the following screen:
11. The subsystem on the target node will display the MQ active jobs.
NOTE
Other active iTERA HA jobs will also be displayed in the iTERA HA
subsystem.

Ending WebSphere MQ Replication

To end WebSphere MQ replication, perform the following steps:
1. Press F17 to end replication. This will turn off IFS and library replication
for MQ directories and will end the jobs on the target node.
2. Press F18 to end journaling on the MQ IFS directories. Files cannot be in

use (Queue Managers must be inactive). A pop-up window will display
with a verification message: “To end journaling on ALL files you must end
all activity on those files. Proceed with ending journaling?” Respond with
Y to end journaling.
Role Swap Procedures for WebSphere MQ

The iTERA HA Role Swap Checklist contains steps for handling WebSphere
MQ Replication during a role swap.

System Values Replication (5.8)
System Values Replication (5.8)

NOTE
This menu option is only displayed when the component *SYSVAL
has been enabled in the Replication Options screen.
To enable System Values Replication, do the following:
1. Select menu option 30.23.
2. Select F20=Advanced Options.
3. Enter Y for System Values; press Enter.
4. Press F3=Exit.
5. Select option 6=Enable Global on the *SYSVAL option; press Enter.
6. Press F3=Exit twice.
7. Select menu 5.8 to display the System Values Replication screen.
Replicate a system value by selecting option 6=Start Replication.
The system values that have been replicated can be audited individually by
using option 21=Audit or all replicated system values may be audited by using
F20=Audit All. SYSV_AUDP automatically runs in the Audit Command
Console and audits all replicated system values.
Entries on this screen in red text (default) are not eligible for replication. This
is also indicated in the Allow Repl column.

Work with Output Queue – E2WRKOUTQ (5.22)

The E2WRKOUTQ command allows the user to sort, search and filter spool
files based on specific criteria.
If Enter is pressed without specifying an output queue, the Work with All
Output Queues screen is displayed.
Selecting option 5 for a queue displays the Work with Output Queue screen,
where you may perform a variety of tasks on the various output queues listed.

Monitoring Your System
6
Monitoring Your Nodes

One of the most vital functions of iTERA HA is accurately verifying that
data in the target node is fully synchronized with the primary node. Because
of the extensive automation incorporated into iTERA HA, this can be
typically done by an operator in less than an hour a day. Additionally, iTERA
HA presents all vital information on a few simple-to-understand screens so
that any replication issues can be quickly viewed and investigated.
It is uncommon that an object on a target node would lose synchronization

with the same file on the primary node. If this does occur, iTERA HA will
usually resynchronize it before it is ever detected that a problem existed.
System Monitor
The System Monitor displays the status of the iTERA HA mirroring process.
From this screen, use F16 (Mirror Process Monitor) to easily navigate to
additional mirroring information and even manage journaling components.
• From the Main Menu select option 1 – iTERA HA Monitoring Menu.

Select option 1 – System Monitor.
• From any iTERA HA command line, type E2SYSMON.
NOTE
When displaying the System Monitor, it is important to remember
that the data is current as of the last time the node updated the
data via the system monitor job. The system monitor job regularly
refreshes this information every fifteen minutes. To manually
update the data, press F10 from the primary node.

Chapter 6: Monitoring Your System
NOTE
Only selected fields are described in the table below. Consult the
Reference Guide for additional content.
Selected Field Description
% Total Disk Storage See “Troubleshooting High Disk Space” on page 310.
Used
% Total Used By
Receivers
Displays the overall status of the Role Swap Readiness

Monitor, which can be accessed from the F14 key on
this screen, 1.7 from the main menu or 40.7 from the
Role Swap Processing menu.
Role Swap Readiness The overall status of the Role Swap Readiness Monitor
must be OK in order to safely perform a role swap.
See “Role Swap Readiness Monitor (1.1 F14 or 1.7)”
on page 152 and “Role Swap Readiness Monitor Issue
Resolution” on page 285.

System Monitor
Displays for each node whether all journals are active

or inactive. Under the primary node, there are two sta-
tus fields (one for local, the other for remote journals)
that may contain either YES or NO in each.
Under each target node there is a single status
(YES/NO) field.
The status indicates whether all journals are active. If
the status field displays a value of *NO in any column,
it means that one or more journals are inactive. To see
a list of all journals, press F4 to access the Mirroring
Process Monitor screen (1.1, F16). See below.
Local/Remote Journals The first YES/NO indicator for the primary
Active corresponds to the local journal on the primary node,
and indicates whether all journals are actively
journaling files and entries are being put into its
corresponding journal receivers.
The second YES/NO indicator for the primary
indicates the status of all remote journals on the
primary node, which is actually the “send” portion of
the remote journal that bridges the primary node with
the target node(s).
The YES/NO indicator for the target node indicates
the status of the “receive” portion of the remote
journal on the target node.
Indicates whether all apply jobs are active on the target

node. A status of “*NO” will be displayed if any of the
Apply Jobs Active apply jobs are inactive for any one remote journal pro-
cess. To view the individual apply jobs, sign on to the
target Node, and then select 3.4.
Network - Indicates whether it can be “pinged” from

the primary node using the iTERA HA IP address.
Subsystem - Indicates whether whether the MONSTS
job on the target is able to communicate back to the
primary. If the status is NO, then one or more of the
following is occurring:
Network/Subsystem
Active 1. There is a communications issue. (Use 30.7 to test
communications and resolve any communications
issues that exist.)
2. The E2JOBQ is held. (Release the job queue.)
3. The subsystem on the target is not active. (Start the
subsystem on the target.)

Any number the Objects Requesting Sync field

indicates exposure on the target node. iTERA HA
Objects Requesting detects objects that need synchronization during the
Sync mirroring process and when an audit process runs.
iTERA HA will normally automatically resync the
objects. (Press F6 to view the objects.)
IMPORTANT
Press F10=Update Monitor on the primary in order to view the most
current data.
IMPORTANT
Any NO indicators for Local/Remote Journals Active, Apply Jobs
Active, or Network/Subsystem Active fields need to be investigated
and resolved.
How to Change the System Monitor Refresh Time

Interval
1. Select Work with Monitored Jobs (2.21).
2. Press F8=Set Delay to set the System Monitor delay time.
3. Enter the desired time for the System Monitor delay interval.
4. End and restart the iTERA HA Subsystems on both nodes for the
change to take effect.
a. E2ENDSBS - primary first, then target.
b. E2STRSBS - target first, then primary.

System Monitor
Mirroring Process Monitor (1.1 F16)

The Mirror Process Monitor displays the current state of iTERA HA
components such as Remote Jrn Send Status, Exposed Entries, and Apply Pending.
Access the Mirror Process Monitor from the System Monitor on the primary
node: 1.1, F16.
Item Description
Displays the number of journal entries that have not

Exposed Entries
been sent from the primary node to the target node.
Displays the number of journal entries received by the

Apply Pending journal receivers on the target node that have not yet
been applied to the objects.
Specifies whether the apply Job process is active on

the target node. The values in this field are usually
either active (the job is in a time wait or a run status),
Apply Job Status
or N/A (job is not active). However, if a different
status code appears, it means there is either an error or
a delay.
F14=Main Journals Same as option 3.1.
F7=Journal Receivers Same as option 3.2.
F8=Remote Journals Same as option 3.3.
F9=Apply Jobs Same as option 3.4.

Objects Requesting Sync (1.1 F6)

The Objects Requesting Sync screen is accessed using the F6 function key from
the System Monitor.
Field Description
The source library and object need to be synced to the

Library/Object
target node.
Identifies the reason for syncing the object; e.g.,

Object Attribute *FILE-PF indicates a physical file is starting to be
journaled.
The iTERA HA v6.0 Reference Guide contains

Reason descriptions for values that could be displayed in this
field.
If the object to be synced size is equal to or less than

the Network Max Sync Size, then the object will be
synced over the network. If not, then selecting F20
Size
will initiate a sync using a tape. The Network Max
Sync Size can be increased, if necessary, by selecting
F19.
Indicates the type of sync, such as database relation-

Sync Type
ship (physical and logicals) sync.
For additional information on Objects Requesting Sync, see “Objects

Requesting Sync (1.1 F6)” on page 150 (in the Monitoring Checklist).
iTERA HA Message Log (1.1 F11 or E2MSGLOG)

All iTERA HA messages are contained external message files. Both first-and
second-level text is available for most screens. The detailed descriptions are
intended to be able to help you easily determine the corrective action for
system issues that may arise.

System Monitor
Select option 1 for a message to view second-level help text.
Option 1 for Message ID HAE0175 displays the following:
• When F6=Work with App is selected the iTERA HA program that is most
likely to be used to resolve the issue is executed. F6 is the same as selecting
option 5 from the main iTERA HA Message Log screen. The program
name is indicated in the “Solution” line toward the bottom of the screen.
– F6/option 5 for this particular example would display the iTERA HA

subsystem, where the A1_SNC_N03 job would be in MSGW status.
– To illustrate another example, F6/option 5 on a message that indicates

an object cannot be saved would display the 4.21 Mirrored Object
Maintenance screen, where further investigation could be done.
• F7=Work with Solution is the same as selecting option 7 from the main
iTERA HA Message Log screen. It executes the solution most likely to be
used to resolve the issue.
– For example, if F7/option 7 is selected for a message that indicates an

object could not be saved due to a lock on the object, it would execute

IBM’s WRKOBJLCK command, where further investigation could be

done.
• F8=Send Message is the same as option 6 from the main iTERA HA

Message Log screen. The selected message is sent to the designated message
queue.
• Use F20=Alternate Message Help to key in environment-specific data or

instructions about the message. For example, in some customer
environments, a remote journal receiver may become detached after a
housekeeping routine is run or after experiencing a temporary interruption
in communications. A typical corrective action would be to manually
activate the receiver. After the issue is resolved, select option 1 on the
message, then F20, then key in the instructions an operator should execute
in order to resolve the issue in the future.
Message Routing and Message Groups

Messages can be defined to be sent to a message queue, iTERA Alert, the
message log, or to any combination of the three. This is done by selecting
F8=Message Routing from the main message log screen, then option
2=Change for the desired message. The following screen is displayed:
“Message Groups” can be defined for sending a message to more than one
message queue. (The default routing for all messages is to the message log
only.)
Role Swap Readiness Monitor (1.1 F14 or 1.7)

The Role Swap Readiness Monitor automates many of the tasks and tests that
are required prior to a role swap. The overall status of the Role Swap Readiness
Monitor is displayed in the System Monitor.
Access the Role Swap Readiness Monitor screen from:
• 1.7 (Monitoring menu)

System Monitor
• 1.1, F14 (System Monitor)

• 40.7 (Role Swap Processing menu)
While many of the tests executed can be run individually, the monitor provides
a simple, streamlined, automated process from which to manage them. It also
ensures that no tests are skipped nor forgotten.
The tests executed by the Role Swap Readiness Monitor are summarized in the
table below (listed alphabetically). For issue resolution information, see “Role
Swap Readiness Monitor Issue Resolution” on page 285. Additional
information on the monitor is located in the iTERA HA v6.0 Reference
Guide.
Test Description Details Option 1 Behavior
Data Apply Jobs Checks the apply jobs for the data Displays the
APPLYDATA journals to ensure they are active on the Mirroring Process
target machine. Monitor. (1.1, F4).
Check the results by
Apply job for the IFS Checks the apply jobs used for IFS
APPLYIFS transport journal Replication to ensure they are active on
viewing the Remote
the target machine. Journal screen (F7
twice).
Apply job for the MQ Checks the apply jobs used for MQSeries
APPLYMQ transport journal Replication to ensure they are active on
the target machine.
Apply job for the Checks the apply jobs for the transport
APPLYTRAN transport journal journals to ensure they are active on the
target machine.
Audit Monitor Displays the status of the Audit Displays the Audit
Command Console. All audits must have Command Console.
AUDIT_MON successfully run within their designated
run intervals in order to indicate an OK
status.

DDM Test The test creates and updates a DDM file Displays the DDM
to the target machine and then creates a Commands menu.
DDM
command to send one from the target to
the primary.
Device Configuration This test checks the replication of Displays the

Replication configuration of objects such as devices, Configuration
controllers, and lines. This component Replication screen.
DEV uses the transport method.
This test is only available if the
component is toggled on in the
Replication Options screen (30.23).
Directory Entry Directory Entry Replication (including Displays the

Replication SMTP information) uses the transport Directory Entry
method. Replication screen.
DIRE
Encryption Test The encryption test checks to ensure the Displays the Work
licensed programs needed for encryption with License
ENCRYPTION exist on the machine. Encryption Information screen.
programs are needed for User Profile
Replication.
FTP Test The FTP test creates a file, FTPs it to the Displays the
target node, then issues a command to do Configure TCP/IP
the same from the target node back to the FTP screen.
primary. Additional
FTP
information on this
IBM screen is
accessible via the F1
key. command.
Heal Test Makes sure that there is not a backlog of Option 1 for this test
heal records that have not been applied. is only valid on the
target node and
HEAL
displays the Heal
Status Summary
Search screen.

System Monitor
Integrated File System The Integrated File System data can be N/A
Test replicated. They are copied using their
own journals which are separate from
data stored in libraries.
This test verifies that IFS Replication is
current. If the number of unprocessed
IFS
entries exceeds the limit an error status is
displayed. The warning threshold can be
adjusted.
IP Connection Checks through all IP mappings defined Invokes the iTERA

Warnings Test to iTERA HA (Takeover IP address) and HA TCP/IP Interface
IPCONWRN ensures they are functional and in the Mapping program
appropriate state. A failed IP test will (30.22).
create an entry in the message log.
iTERA Alert This test is only available if iTERA Alert N/A

is installed on the system and if the
ITALERT
iTERA Alert is a separate product offered
by Vision Solutions.
There will be one Checks that all key jobqs and subsystems Displays a window
JobQ test for each are active (user-defined as well as Vision which allows access to
JobQ. Solutions-recommended). the specified jobq,
JOBQ If the subsystem is not active, the test will subsystem, or both.
not run. JobQs are critical in role swaps.
They are attached to the subsystem so the
subsystem must be active.
Job Scheduler Sent via the transport method. Displays the Job
This test is only available if the Scheduler Replication
JOBSCDE screen.
Library Sync Status Checks for any libraries that have had the Displays the Work
‘new’ status cleared, have had syncing with Libraries screen
LIBSYNC
cancelled, or have had journaling ended (4.11).
on them.
System Monitor Verifies that all settings in the System Displays the System
Thresholds Monitor are within the limits specified in Monitor Alert Limits
MONTHR
the System Monitor Alert Limits screen. screen
(E2MONTHR).
MQSeries Test This test is only available if the

MQ component is toggled on in the

Object Monitor Tests OBJMON1 will display in error if the OBJMON1 displays
MONTHR test is in error. the System Object
OBJMON2 and OBJMON 3 tests verify Monitor.
OBJMON1
through that the jobs are running and not falling OBJMON2 and
OBJMON3 behind. OBJMON3 display
the Object Audit
Level Definition
screen.
Object Sync Status The test checks for objects that are not in Invokes the Mirrored
OBJSNCSTS a normal journal, sync, or omit status in Object Maintenance
the 4.21 screen. program (4.21).
Object Sync Requests Checks for any objects currently awaiting Displays the Objects
synchronization. Requesting Sync
screen (1.1, F6) in the
OSR context of primary or
target. If viewing the
target node data, press
F7 twice.
Ping Test Performs the Ping test. Displays the

PING Configure TCP/IP
menu (30.2).
Record Count Audit Checks objects with record or member When option 1 is
Test count differences reported in the 1.21 selected from the
screen. target node, the
RCDCNT
This test runs only on the target node. If Record Count Audit
it fails, you must resolve the issue from Maintenance screen is
the target node. displayed.
Remote Journal Status Checks the status of remote journaling. Displays the Remote
RJSTS Journal Maintenance
screen (3.3).
Remote Commands Some commands are sent to another N/A

RMTCMD system in the group. These commands are
process by the remote processing job.
Spool Files *OUTQs and reports can be replicated Displays the Output
using this process. The spool file Queue Search screen.
replication uses the transport method.
Reports sent between boxes running
SPLF V5R4 is different than from older
systems.

System Monitor
Remote SQL Test Makes sure that the remote SQL process Displays the
SQL works and is functioning as desired. Configure TCP/IP
screen.
System Monitor Displays the status of the iTERA HA N/A

System Monitor (1.1). All values (except
for the status of the Role Swap Readiness
SYSMON Monitor) must be within the acceptable
limits designated in the System Monitor
Threshold Settings screen
(E2MONTHR).
System Values The system values are evaluated to ensure Executes IBM’s
SYSVAL they are set correctly. WRKSYSVAL
command.
Tape Media Library This test is only available if the

TAPMLB Support component is toggled on in the
User Profiles This test is only available if the Displays the User
USRPRF component is toggled on in the Profile Search screen.
The Role Swap Readiness Monitor tests are initiated via the xx_RSRMON job
on all nodes. The monitor on each node will display all tests appropriate to the
machine and the status of each test.
The Summary Status field at the top of the screen will indicate OK if all tests
have completed successfully within the past thirty minutes.
• The WRN status (warning) is displayed if at least one test has issues that
should be investigated but it may not be critical to resolve prior to
performing a role swap.
• The ERR status is displayed if at least one job is in error status. Also, if a
mandatory test is in ERR status, it will be displayed in red. Errors should
be investigated and resolved prior to performing a role swap.
Jobs that have not run within the past thirty minutes will be displayed in
reverse image in the status column (for both Local and Remote nodes). Any
tests that have not have finished processing are indicated by two plus signs (++)
or two asterisks (**), displayed in the Status column. Two dashes are displayed
for tests that do not run on that node. When the test status is displayed in
reverse image (highlighted), it indicates that the test has not been run in the
last thirty minutes and the results may be invalid.
To submit all tests (for both Local and Remote nodes), press F18. To run a
single test interactively, select option 7. If a test is run locally on the primary it

is also submitted on the target node via the SNDRMTCMD, if appropriate.

Option 8 will submit the test to the iTERA HA job queue.
Press F5 periodically to update the display until all results are reported.
Status
Displayed Test Outcome
OK Test successful
ERR Test unsuccessful
WRN Warning (possible error message on batch job)
Types of Tests Performed - Option 2

The following is a summary of the tests that are performed on the various
components. To change the type of test performed, select option 2 on the
component. Any or all of these tests may be ignored by placing a Y in the
ignore field.
Test Description
Replication components typically have a transaction

file that stores all pending changes or requests.
Usually the object monitor records entries for adds,
changes, and deletes. Audits are usually added to this
Transaction file file. Control files and data area changes are also
included. If an error is related to the transaction file, it
is usually because either the processing job is not
functioning correctly (not running, held, *MSGW)
or it is running behind and trying to catch up.
Some replication processes have an error file. This file

is used to allow the user to see what replication
Error file processes failed. If a sufficient number of errors are
found on the target then the user should review the
list on the target system. Not all errors are problems.
Many replication processes use one or more journals.

This test will make sure the journal apply process is
Journal
active for all journals of a specific type. For example,
Spool File Replication uses the transport (T) journal.
Some replication processes use one or more jobs to

replicate. This audit allows up to three primary jobs
Job Test and three target jobs to be validated. Each job is tested
to make sure it is active and does not have any
messages.

System Monitor
Test Description
Some replication processes require that a library be

loaded on the system in order for that component to
Library Test work. This test makes sure the library is loaded. It
does not make sure any license key is loaded, or that
the contents of the library are correct.
A component may use a test program to perform any

Test Program test that is unique to the component that is not part
of the standard processing.
Some components use one or more exit points

(WRKREGINF) to receive notification of changes.
Exit Point An exit point is an IBM break point that allows the
user to be notified of a change such as when a user
profile is changed.
Some components have special tests such as the

Special test *JOBSCD object is set to audit level of *CHANGE.
This makes sure that setup is correctly.


Monitoring Checklist
7
The previous section introduced the audits that are critical to system upkeep.
In this section we discuss the Monitoring Checklist, which is a very
important system procedure that should be executed at least once daily. Since
the goal of high availability software is to ensure that there is an exact
duplicate of all objects selected for replication residing on the target node at
all times, audits must be executed and the system must be monitored on a
regular basis to ensure this occurs. Compromise on regular execution of
either auditing or monitoring and you will be compromising the reliability
of the software to keep objects in perfect sync.
The items on the Monitoring Checklist are divided into three sections:
Daily, Weekly, and Monthly activities. Run through the checklist at
approximately the same time each day, since system resources can fluctuate
throughout the day to help ensure consistency; (for example, “Disk Space
Used” will be much higher during day-end processing than at other times.)
The checklist provides a brief description about each step and space to record
the results. Recording the results for a period of time and periodically
referring to them will provide a baseline for comparison of the results.

Chapter 7: Monitoring Checklist
Daily Tasks Section

Work through the following items to review, troubleshoot, and resolve issues.
NOTE
Some checks can be performed from any node, while others must be
performed specifically on the node indicated. An “N/A” will be
indicated in the PRI or TGT column if a check cannot or should not
be performed on that node. If more than one target node is defined
in your CRG, perform the checks on all target nodes, where
applicable.
Description PRI TGT
Access the System Monitor (1.1, all nodes). Check the iTERA HA subsystems on all nodes (F7) ❏ ❏
and ensure they are active, that all the normal jobs are running, and that no jobs have a status of
MSGW. Press F12 to return to the main System Monitor screen. (Refer to the section “iTERA HA
Subsystems” on page 18 for a description of the jobs that should be running.)
On the primary, press F10 to update the System Monitor. Check for current values in the Last ❏ N/A
Update Time fields (for both primary and target). If any value appears in red, it indicates that it has
been twice as long (as the preset update time interval) since the last System Monitor update. If it
does appear in red, press F10 again. The value should update (and the color should change to
green). If it does not update correctly, verify that the subsystems are active (F7). If not, restart the
subsystems (on all nodes) then return to the System Monitor and press F10 to update (you may
need to update the monitor twice if statuses are not corrected with the first update)
Check disk space usage in the % Total Disk Storage Used fields to make sure you have sufficient ❏
unused disk space and that journal receivers are not using excessive amounts of disk space.
The % Total Used by Receivers typically runs in the 3-5% range, depending on the system and how ❏
long receivers are retained. If greater than that, it indicates that the apply jobs aren’t running
correctly and receivers aren’t being deleted. (Note that you are only checking the primary node, but
you should record the values for both primary and target node usage.)
Verify that all three Local/Remote Journals Active fields display Yes statuses. ❏
If any display *NO, press F16 to access the Mirroring Process Monitor screens on both the primary
and target nodes to view additional information and to access the screens where you can manage
journaling components.
If remote journaling is inactive, then activate it.
Verify that Apply jobs are active. The Apply Jobs Active field should display Yes. If this displays No, ❏
press F16 to access the Mirroring Process Monitor screen on the primary node and check the Apply
Job Status field. To manage the apply job, select 3.4 on the target node.

Daily Tasks Section
Description PRI TGT
Verify that the Network/Subsystem Active field (underneath the Backup 1 node column) display Yes ❏
in both positions. These values indicate whether the systems are able to communicate with each
other.
If the first position value is *NO, do the following:
1. Restart the DDM servers (30.5 on all nodes) then return to the System Monitor and press F10
to update (may need to update twice).
2. Check the status of the xx_RMTCMD job in the iTERA HA subsystem. If the status is MSGW,
answer the message (option 7). If in HLD status, release the job (option 6).
3. Perform the Ping, FTP, DDM check (30.7) on the primary node. Review the E2MSGLOG for
results of the test.
If the second position value is *NO, then it indicates FTP is not active. Perform the Ping, FTP and
DDM Test (30.7) on all nodes, to all nodes.
If these solutions do not work, contact CustomerCare for assistance.
Check Journal Entries Not Applied. If this value is larger than usual, press F16 to access the ❏
Mirroring Process Monitor to determine whether they are exposed or applies pending. Exposed
entries are related to communications or to a job generating more transactions that can be handled
by the communication lines. Contact CustomerCare for assistance with fine-tuning this.
Review Objects Requesting Sync on the primary node (1.1, F6) to see if any files are waiting to be ❏
resynchronized. The number of objects requesting resynchronization will appear in the Objects
Requesting Sync field.
If there are excessive objects requesting sync, then press F16=Submit Extra Network Sync as needed
to submit additional sync job(s).
Normally, the system will automatically resynchronize necessary objects. However, you should
verify the following:
1. Review the reason for the objects requesting sync.
2. Review the status; verify that the system is in the process of resyncing.
3. If an object is not resyncing, then on the primary, review the size of the object and verify that it
is smaller than the Max Network Sync size. If not, you will need to either increase the Network
Max Sync Size (F19) or perform a tape sync (F20).
Review the Object Sync History Log (1.1 F6, F17 on the target). Review the history of all objects ❏
requesting resync.
Once issues have been resolved, on the Object Sync History Log screen clear the log by pressing
F20=Clear Log.
Review iTERA HA system messages on all nodes (1.1 F11 or E2MSGLOG). Review messages and ❏ ❏
resolve issues, as needed.
Once messages are resolved (on all nodes), clear the message logs (F16=Clear Info Messages). This
will move all messages to the history log (F18=History). Clear the History Log, as needed, using
F16=Clear Info Messages.

Description PRI TGT
Role Swap Readiness Monitor (1.1 F14 or 1.7, all nodes) ❏ ❏

Verify the Summary Status field of the Role Swap Readiness Monitor is OK/OK. If not, press F18
to submit all tests.
Review the results of the Role Swap Readiness Monitor. If any tests indicate a status of ER,
investigate and resolve as necessary (see “Role Swap Readiness Monitor (1.1 F14 or 1.7)” on
page 152, for more information). Complete the rest of the monitoring checklist, then resubmit the
Role Swap Readiness Monitor tests and verify all tests are in OK status.
Issues encountered should be investigated and resolved on the node that is displaying the error.
For troubleshooting steps for the OBJSNCSTS test, see “Mirrored Object Check (Option 1 on the
OBJSNCSTS test in the Role Swap Readiness Monitor)” on page 170.

Daily Tasks Section
Description PRI TGT
Record Count Check - On the target node, select option 1 on RCDCNT test in the Role Swap ❏
Readiness Monitor or 1.22 to display the Record Count Audit Maintenance screen.
Set the Filtering I/O field to “O”, enter an asterisk “*” in each of the Alloc State and RSync Y/N
fields, then press Enter.
The Allocated Status column displays for each object whether the object could be allocated.
Objects with Y indicate that the record count audit was able to obtain an exclusive lock on the
object, indicating that no other process was changing, deleting, or adding records to the object.
Therefore, the count obtained was accurate. Objects with N indicate that another process has the
object allocated, meaning that adds, deletes, and changes may be occurring. Therefore, the record
count may or may not be accurate. Errors indicated on these objects may or may not be true errors.
The objects should be reaudited.
A blank in the Audit Error column will omit any records that completed normally and will display
only those records that had an error.
The Audit Error column displays for each object whether a record count error was found. If the
field is blank, there was no error for the object. The accuracy of the error is dependent on whether
or not an exclusive lock was obtained. If the Alloc State field is N, then the errors may not be
correct and a reaudit of the object is recommended.
Error types are as follows:
• E – Indicates a count error. The variance between the objects is shown in the Audit Variance
column.
• M – Indicates a member audit error (a disparity was found in the number of object members).
The variance is shown in the Audit Variance column.
• I – Incomplete IBM API info error when the audit was done.
• F – Error in record format.
• H – Indicates that the object is missing records. It should either be resynced or a CRC audit
should be run over the object which will initiate the Heal process.
• + – Data Queue has more entries on the primary node.
• - – Data Queue has more entries on the target node.
• [blank] – Indicates no errors.
For items that display errors, perform the following:
1. Select option 8=Audit (may select several items with errors simultaneously) to re-audit. For
errors with member differences, use option 9=Member Audit, then option 6=Resync.
2. Wait a few moments then press F5 to refresh the screen.
3. If the same objects reappear, select option 6 to resync the objects.
Clear the Record Count Audit screen (F23).

Description PRI TGT
Review the Object Attribute Audit results on the target node (1.23 and E2MSGLOG) ❏
Messages generated by the Object Attribute Audit will be displayed in the iTERA HA Message log,
which you should have reviewed earlier. If not, review them now.
On the target node, select 1.23, then select option 5 for the most recent audit (top of the list),
continue to select option 5 to drill successively down into the object level. Manually resubmit the
audit for any objects with attribute discrepancies.
Object Monitor Check (1.4, primary) ❏

Press F5=Refresh or F19=Start Auto Refresh. The Object Monitor is an indicator of whether
iTERA HA is keeping up with the number of entries generated in QAUDJRN. It is a snapshot of a
point in processing time.
• Check the Object Monitor’s Process Status (this field displays the status of the OBJMON job).
• If the Present Delay field is excessively high (relative to your system), indicate the number. (A
processing delay does not necessarily indicate a problem. If the processing difference is excessive,
investigate further.
• The Process Status field typically displays a status of Held/MsgW, which indicates that it is
waiting for something to process (although the Held/MsgW status may seem misleading, in this
case it doesn’t indicate there is a problem). While it is processing, it will display the status of
Current. However, if it displays Not Current, do not be concerned (just ignore the status for
now).
• The Delay field under the Start section is the number of entries iTERA HA was behind
QAUDJRN at the moment the Object Monitor was accessed. Some delay here is normal
because 1:1 processing is virtually impossible.
• The Delay field under the Present section is the number of entries iTERA HA was behind as of
the most recent refresh of the data. Again, some delay here is normal. If the Present Delay rate is
significantly larger than what is normally displayed for your system press F5 to refresh the screen
every few moments. If the number does not decrease over time, make note for further
investigation.
• Review the Processing Rate. The processing rate for iTERA HA should be greater than that for
QAUDJRN. If not, make sure the xx_OBJMON job in the subsystem is running. You may need
to adjust the job’s timeslice and/or priority (check with your System Administrator if you need
to know how to do this).

Daily Tasks Section
Description PRI TGT
The IFSMON information in the lower portion of the System Object Monitor screen (1.4, ❏
primary) provides details on how well the xx_IFSMON job is processing requests. It is especially
helpful for those doing a significant amount of object-level replication by helping to determine
whether additional jobs should be run.
The Process Difference field displays the difference between the numbers in the Previous to Process
and Remain to Process fields. If the number is negative, more requests have been processed than
there are waiting to be processed. If positive, the xx_IFSMON job is not keeping up with the
number of requests. You may need to adjust the timeslice and/or priority for the job.
Spool File Monitor (1.6 primary) ❏

If replicating output queues and spool files, check the Spool File Monitor for the number of
unprocessed entries.
IFS Audit Review (If you replicate IFS, perform this monitoring check.) ❏ ❏
• Access 1.8 on the primary. If the IFS_AUDIT test is beyond the recommended run time interval
manually execute it using option 8=Submit.
• Access 1.8 on the target. Select option 1 on IFS_REVIEW to view the Display IFS Audit
History screen.
View the audit details by selecting option 1 in conjunction with the appropriate Fkey. In the
column to the right, indicate the numbers for the following:
– Authority Failures (opt 1 w/F8 [Enter], then opt 5 on any entry listed)
– Attribute Failures (opt 1 w/F9 [Enter], then opt 5 on any entry listed)
– Data Failures (opt 1 w/F10 [Enter], then opt 5 on any entry listed)
NOTE
If replicating QDLS, then the first time option 1 and the appropriate Fkey is selected,
leave the Skip QDLS Paths field blank. The QDLS paths are automatically built. These
paths establish the necessary links to be able to view the QDLS objects within the audits.
It typically takes longer to compile the audit data when building these paths. Once the
QDLS path information has been built, a Y will automatically be placed in the field and
subsequent use of option 1 will skip the build of the QDLS object paths. If you need to
rebuild the QDLS path information, change the Y to N or blank. If not replicating
QDLS, enter Y in the Skip QDLS Paths field.
Heal Status (3.7, target) ❏

Check the number of pending heal requests. Investigate and resolve.
In Non-Mirrored Object Replication Check (4.30, primary), verify that all objects that need to be ❏
replicated are being replicated. The Last Sync Date/Time data should be current. (This may be
scheduled in a job scheduler to run daily.)

Weekly Tasks Section
Description PRI TGT
Locked Object List (4.22 – primary)

For objects that do not have current locks, either start journaling and replication opt 9=Sync Object
on them or remove the filter (F15), sync the objects (, F10=Sync Roll Request), verify the objects are
synced (E2MSGLOG on target; MSG ID HAE0339), then remove them from the list (opt 4).
Changes to objects synced via a non-mirrored sync are not replicated. If the object is a file, data area,
or data queue, it is best to start journaling and replication on the objects so that subsequent changes
to the objects are captured in the journal.
Check Data Queues (1.50.1, primary). If not empty, check E2MSGLOG (not necessary if running ❏
V5R4 or higher).
User Profile Component Checker (1.50.2 – Both) ❏ ❏

This option allows you to audit the primary and target nodes for any objects used by User Profiles
that exist on one node but not on the other. It indicates what components are missing on the current
node. The screen that is accessed displays warnings and errors. Warnings are informational only and
will not prevent the user profile from being used in the event of a role swap. Errors indicate critical
problems, which will prevent the user profile from being used after a role swap, unless resolved.
Press F18 on both nodes to load the information. The message “No Records Found” will be
displayed at the bottom of the screen if there are no missing components. Any missing records will
need to be resolved.
Job Description Component Checker (1.50.3; all nodes) ❏ ❏

This option allows you to audit the primary and target nodes for any objects used by Job
Descriptions that exist on one node but not on the other. It shows what components are missing on
the current node. The screen that is accessed displays warnings and errors. Warnings are
informational only and will not prevent the Job Description from being used in the event of a Role
Swap. Errors indicate critical problems, which will prevent the Job Description from being used
after a Role Swap unless resolved.
Press F18 on both nodes to load the information. The message “No Records Found” will be
displayed at the bottom of the screen if there are no missing components. Issues identified by the
audit, such as missing libraries, printers, etc., will need to be resolved as needed (e.g., restore a
library, create a printer device, recompile a program, or delete the job description).
Check for New Libraries (1.50.4; primary) (This feature is also available from 4.11, F7, F7.) ❏ N/A
The status column will display “New” for any libraries that have not been assigned to a journal or
that have been added after iTERA HA was installed. This status prompts you to determine whether
you want to select the library for mirroring. Typically, just use option 21=Quick NetSync libraries. If
you don’t want to mirror the library and want the “New” status removed, use option 16 next to the
desired library and press the Enter key (or press the F21 key to remove the new status from all
libraries). They will no longer be displayed in the new libraries list.
Check Syncing Libraries for Cross-Library Dependencies (1.50.4, F6; primary) ❏ N/A
Review cross-library dependencies.
Check User Profile Replication (1.50.5.1, primary). Press F16=Dft Map and verify the map is ❏
correct. If modifications are needed, make them and then use F21 to quick sync profiles.

Weekly Tasks Section
Description PRI TGT
Check the User Profile Replication Errors screen (5.1, target) for failures and resolve as necessary. ❏
Check for New IFS Objects or Directories (1.50.5.2, opt 5, primary) ❏ N/A
1.50.5.2 displays the IFS Work with File Systems screen (as does 5.2). Select option 5 for one of the
file systems in order to drill down into the file structure and locate new IFS objects and directories.
Objects added to a directory currently being replicated can automatically inherit journaling, if
specified in the IFS Journaling Defaults (5.2, F7). (This should have been defined when IFS was
initially replicated.)
You must drill down into the various subfolders in order to spot new objects that need to be
replicated. For this reason, it may be extremely helpful to maintain a map of replicated directories
and objects, indicating the location of subdirectories and objects that should not be replicated.
Check Spool File Replication (1.50.5.3; primary). Select F7 twice to review new output queues and ❏
replicate as needed.
Check Configuration Replication (1.50.5.4; primary). Select F21=Config List. Review for new ❏
items and replicate as needed. Review Configuration Replication errors (if needed, the objects can
be rebuilt from 5.4 on the target).
Check Job Scheduler Replication (1.50.5.5; primary). Ensure that everything that should be ❏
replicating is. Errors can be fixed through 5.5 on the target.
Check WebSphere MQ Replication (1.50.5.6; primary). If replicating WebSphere MQ, ensure that ❏
all queue managers are replicating.
Check Directory Entry Replication (1.50.5.7; all nodes). Review the maps and any errors. Review ❏ ❏
errors in 5.7 on the target.

Monthly Tasks Section
Description PRI TGT
If a new iTERA HA Service Pack has been released, load and apply the PTFs. ❏ ❏
• Load and apply PTFs for XP (Cross Product) using menu option 10.46 (specifies product ID
7PA2K02; release V4R3M0).
• Load and apply PTFs for iTERA HA using menu option 10.45 (specifies product ID 7PA2K05;
release V6R0M0).
• If using iTERA Alert, load and apply PTFs using menu option 10.47 (specifies product ID
7PA2K25; release V6R0M0).
NOTE
iTERA HA PTF Service Packs are released on a regular basis. E-mail notification is sent to
iTERA HA customers when they become available. (Notify CustomerCare if you are not
receiving these e-mails.)
Always end the iTERA HA subsystems before applying iTERA PTFs.
Load and apply Cross Product PTFs prior to those for iTERA HA.
Verify E2IFSPRGA runs regularly in the job scheduler to purge audit data.
Verify that the non-mirrored library sync command (SYNCNMLIB) is scheduled to run regularly.
Troubleshooting
Mirrored Object Check (Option 1 on the

OBJSNCSTS test in the Role Swap Readiness
Monitor)
Access the Work with Mirrored Objects screen on both nodes and perform the
following:
Set the filter as indicated and press Enter:
This filter will display all objects for which journaling is not active. An object
that displays an “X” in the JRN column (the first filter column) indicates that it
has been manually omitted and you may disregard it. Any object that appears
with a value other than X needs to be investigated in order to determine why it
is not being journaled.

Troubleshooting
This filter will display all objects that are not being synchronized by the system.
For any objects displayed, verify that they should not be synced (i.e., that they
were omitted from synchronization on purpose).
This filter will show all objects that are marked to be omitted. For any objects
displayed, verify that the objects should be omitted. (You may want to set up
an audit exclusion filter to exclude them from the audit so that the objects
don’t appear in this list on a daily basis. To do so from this screen, press F9,
then F6.)


Auditing
8
Audit Command Console

The Audit Command Console is a central location from which to monitor
and maintain all of the iTERA HA audits. Each audit is defined with a
minimum interval time frame in which, if the audit hasn’t run, the Audit
Command Console initiates it. To ensure that audits run at the
recommended frequency and that they do not overlap, a setting exists for
specifying an audit to run at a preferred time. Alternatively, audits may be
scheduled in a job scheduler.
The Audit Command Console also functions as a safety net. Each audit in
the console has a setting that controls the maximum amount of time that
may pass before it is automatically initiated. Once the audit has completed
(whether initiated by the console or from a scheduled job), the results are
reported back to the console. If, for some reason, the audit does not run
within the recommended interval, the status indicator will flash.
If an audit does not run within the maximum allowed run time, an alert is
generated by the console and, depending on the audit and the circumstances
or severity of the alert, is sent either to a message queue, iTERA Alert, or to a
designated messaging application. If sent to iTERA Alert (a separate message
queue monitoring application), a system operator can be notified by e-mail,
text message, message queue, or all three. iTERA Alert documentation is
located in the iTERA HA v6.0 Advanced Features Guide.
Alert intervals are user-defined but may not exceed the maximum allowable
time in which an audit must run. For example, an audit may be set to issue a
standard alert if it has not run in 24 hours and a severe alert if the audit has
not run in 48 hours. The user may lower the time interval in order to be
alerted sooner rather than later, but may not extend the interval beyond the
maximum.
The AUDIT_MON test in the Role Swap Readiness Monitor (1.7) checks
the Audit Command Console to verify whether it is active and whether all
audits have run within their defined intervals. The results of the
AUDIT_MON test are then reported to the System Monitor (1.1). If any
audit is past the Audit Interval, an “ERR” status is displayed. Any warnings
or errors for the Audit Command Console test should be investigated and

Chapter 8: Auditing
resolved promptly. Once issues are resolved within the Audit Command
Console and the test within the Role Swap Readiness Monitor has been
reprocessed, the status for the test will indicate OK for both nodes.
The Audit Command Console can be accessed via a number of ways:
– Menu 1.8
– Menu 6
– 1.7, option 1 on AUDIT_MON
Upon first entering the console, the audits are displayed in a hierarchical order
in relation to the alert priority, but the Auto Audit feature will be turned off.
The initial screen of the Audit Command Console will display all audits
applicable to the node.
NOTE
The audits displayed in the console will vary, depending on
replication options. The default view is to display only active
(applicable) audits (i.e., audits for current replication options that
have been activated). If needed, the entire list of audits is available via
F8=List All Audits.
The following table describes some of the main features of the main Audit
Command Console screen.
NOTE
The following table does not contain descriptions for all options,
fields, and functions available on this screen; consult the iTERA HA
v6.0 Reference Guide for additional details.

Item Description
Auto Audit When Auto Audit is set to *ON, audits in the console will be
executed based on the Audit Interval, which can be defined in the
option 2 screen for the audit. (Audits that have exceeded the
specified run interval display in red in the Current column. The
auto audit feature should remain *OFF until initial syncing is
complete, all audits have been assigned a preferred run time, and
have run at least once.
Sort The sort order can be switched between Priority and Audit
(alphabetically by audit name) using F9. Priority is calculated
based on the Sts (status) column (jobs with errors appear at the
top, followed by jobs currently running), then by the Next Audit
Submission column data. Sorting by Audit sorts the list
alphabetically by audit name.
List Press F8 to toggle between Active and All Audits. The default
view is Active. In the All Audits view, audits for components not
being replicated or audits not applicable to that node are
displayed in red text.
1=Work with Invokes the program that will most likely be used to resolve an
App issue. Where option 1 is not valid for an audit, a message will be
displayed at the bottom of the screen.
2=Change Run status results, frequency settings, and scheduling options are
accessed via option 2. This screen is described in more detail
“Option 2=Change” on page 178.
8=Submit Sends the audit to the job queue for processing. The job will use
the parameters defined in the option 2=Change screen.
Audit Audits applicable to the current node are displayed in green text
(default). When F8=List All Audits is pressed, all audits are
displayed. The audits not applicable to the current node and
audits for components not being replicated are displayed in red.
Audit names ending with P run only on the primary and most
audits with names ending with the letter T are typically for
viewing the results of the audit (on the target). (A corresponding
audit with the same name exists on the target node, but is not
applicable to that node and is displayed in red.)
System Role Indicates the system on which the audit runs, e.g., Source, Target,
or Both. Use F8 to toggle between active audits and all audits.
When viewing active audits, only those audits applicable to the
node you are on are displayed. For example, when on the primary
system, only audits with Source and Both are displayed in the
System Role field.

Chapter 8: Auditing
Item Description
Audit Interval Each audit has a defined maximum run interval. Audits can be
(Type, adjusted to run more frequently than the maximum, but not less.
Current, If the audit exceeds the defined run interval, it will be displayed in
Maximum) red text. The audit interval can be adjusted using option 2.
• Type - Correlates with the value in the Maximum field. For
example, if 24 is specified in the Maximum field and HRS is
specified in Type, the audit will run daily, every twenty-four
hours.
• Current - Possible values include:
– Zero (0) in the default text color if the audit has run within
the number of days or hours indicated in the
Maximum/Type fields. Will display in red text if the audit is
currently running.
– Displays in red text the number of hours or days that the
audit has exceeded its maximum Run Interval.
– Values flashing in red text have exceeded the Alert Interval
specified in option 2.
• Maximum - Correlates with the Run Interval field accessed via
option 2.

Item Description
Next Audit Indicates the date the audit will next be initiated. The date is
Submission calculated by adding to the date the audit last ran the number of
hours and/or days specified for the Run Interval.
Dashes in the Next Audit Submission field on the target indicate
that the audit does not run on that node, but the audit results are
available by selecting option 1.
Sts Status is a combination of the urgency (reflected in the color

coded message) and the text displayed. The four default colors
(red, yellow, blue and green) indicate the relative level of urgency.
The text reflects the status of the job.
Text color indicators:
– Red - Critical - A job has exceeded the specified Alert
Interval.
– Yellow - A warning that indicates there may be an issue that
needs to be addressed.
– Green - The default status.
Text indicators:
– ALERT - If the audit has not been run within the time
specified in the Warning Alert field in the option 2=Change
screen, then ALERT is displayed in yellow text. If the audit
has not been run within the time specified in the Severe Alert
field, then ALERT is displayed in red text.
– Cancel - The job did not complete.
– Done - The job completed normally.
– Error - The job is in error.
– HOLD - The job is in a Held status.
– Init - The audit has never been run or it has been reset.
– JobQ - The job is in the queue, waiting to run.
– Msgw - The job is in MSGW status in the job queue.
– Reset - Displayed after a role swap has been performed.
– Run - The job is currently being executed.
– Submit - The job has been submitted.
– Warn - There are warnings on the job that may need to be
investigated and resolved.

Chapter 8: Auditing
Option 2=Change
Several parameters, such as run status, frequency settings, and scheduling options,
are accessed via option 2.
NOTE
The following table does not contain descriptions for all options,
fields, and functions available on this screen; consult the iTERA HA
v6.0 Reference Guide for additional details.

Item Description
Retain History Indicates the number of days the audit history will be
retained. View audit history using option 5 from the main
console screen.
Submit Window, The audit will only submit automatically if there is at least
Execution Days, one Y in the Execution Days field and will only execute
and Preferred Time during the Submit Window time frame.
Specify a time in Preferred Time if the audit should be
executed at a particular time. It must fall within the Submit
Window and Execution Days in order to be executed.
“0:00 - 0:00” in the Submit Window field (and “0:00” in
Preferred Time) indicates the audit will be executed based on
the Last Run Start time and Run Interval specified.
For all audits except the Object Attribute Audit, the audit
will be submitted in the window and will continue to process
until finished. For the Object Attribute Audit, the audit will
submit and run only during the time length indicated in start
and end time fields. If the audit has not finished processing,
the job is canceled and the next audit run will start with the
library that was being processed when the audit was
canceled. This ensures that all libraries on the system are
audited even when the time allowed to run each day may be
limited. When checking results, you may need to check more
than one audit result entry.
Audit Execution • Interval Type - Indicates hours or days and correlates to

Frequency fields the Run Interval field.
• Current - Indicates the number of interval increments
(e.g., minutes, hours, etc.) that have elapsed since the
audit was last executed.
• Run Interval (Current/Maximum)- Indicates the number
of intervals (hours or days) that must elapse before the
audit will next be submitted. The run interval may be
modified to occur more frequently, but may not be
extended beyond the maximum indicated to the right of
the field value. Default and maximum run intervals vary
based on the audit and are defined by Vision Solutions.
If the audit has not completed within the specified Run
Interval, the value in the Current field on this screen and on
the main console will display in red. If the audit exceeds the
interval indicated in the Severe field, it will flash red.

Chapter 8: Auditing
Item Description
F7=Create Job Displays the IBM Job Schedule Entry screen for the audit
with the minimum recommended run interval settings
pre-populated. Scheduling the audits in the job scheduler is
recommended and helps ensure the audits run precisely at
the same time each interval. While the settings may be
changed, Vision Solutions does not recommend extending
the run frequency beyond the default setting. For example,
you may schedule a job to run more frequently than what has
been programmed in the default, but do not schedule it to
run less frequently.
In order to prevent the audit from being submitted from
both the Audit Command Console and the job scheduler
simultaneously, the scheduled time for the job schedule entry
should be earlier than the time indicated in the Next Sbm
Time field in the option 2 screen.
Do not change the name of the scheduled job. Otherwise,
results for the audit will not be reported back to the console.
F8=View Job Displays the job schedule entry in the IBM Job Scheduler.
Additional audit details are documented in “Audit Detail” on page 185.)
Managing the Audits

Audits will be automatically initiated if Auto Audit is enabled and the audit
reaches the Next Audit Submission date and time. The Preferred Time field in
the option 2=Change screen can be populated to have the audit execute at a
designated time. However, it is best to schedule the audits in a job scheduler in
order to ensure that certain audits do not overlap, to ensure that they run in
the correct order, and to precisely control the day, time, and frequency an audit
is run.
IMPORTANT
When scheduling the following audits, ensure there is no run time
overlap. Allow enough time for each audit to complete before the
next one is allowed to start. They should not run concurrent with
each other.
- AUDSTREAM
- OWNER_AUTH
- DBR_AUDIT
- OBJATRAUD

Managing the Audits
IMPORTANT
The AUDSTREAM audit runs the following audits:
- CHKOBJMTCH
- JRNOBJJRN
- JRNOBJLST
- RCTCNTALL
- RCTCNTCHG
The AUDSTREAM program runs the RCTCNTALL audit on

Saturday and RCTCNTCHG all other days.
When scheduling the AUDSTREAM audit, there is no need to

schedule these individual audits.
When the AUDSTREAM audit runs, the status for each of the
individual audits will be updated in the console.
Special configuration is required for the CHKOBJMTCH audit in

environments with multiple target nodes. Contact CustomerCare for
details.
Scheduled Audits
Vision recommends that all audits be scheduled in a job scheduler. However,
the Preferred Time can be used in the option 2 screen. At minimum, schedule
the following in a job scheduler.
Primary node
Audit name as it
Audit Description should appear in Frequency Schedule Schedule
Day Time
the Job Scheduler
AUDSTREAM Audit Stream HAxxASTRM *WEEKLY *ALL 0100
Database HAxxAUDDBR *WEEKLY *SAT 1400

DBR_AUDIT
Relations
Ownership and HAxxOBAUTH *WEEKLY *ALL 1400

OWNER_AUTH
Authority
ALVL_AUDIT Auditing level HAxxAAUDLV *WEEKLY *SUN 1500
IFS_AUDIT IFS Audit HAxxAUDIFS *WEEKLY *ALL 0100
User Profile HAxxAUSER *MONTHLY *SUN 1600

USRPRF_AUD (Specify 1 for
Relative day of
month parm)
TRG_AUDP Trigger Audit HAxxTRGP *WEEKLY *SUN 1700

Chapter 8: Auditing
Target node
Audit name as it
Schedule Schedule
Audit Description should appear in Frequency
Day Time
the Job Scheduler
ALVL_AUDIT Auditing level HAxxAAUDLV *WEEKLY *SUN 1500
NOTE
Do not attempt to schedule an inapplicable audit (the audit name
will be displayed in red text). Inapplicable audits are only displayed
in the console when the F8=List All Audits key has been pressed.
NOTE
The appendix “iTERA HA Scheduled Audits and Other Jobs” on
page 263 provides a list of all audits in the console. We recommend
that you refer to it when working through the instructions in this
section.
How to Create a Job Schedule Entry for an Audit

To create a job scheduled entry for an audit in the console, do the following:
1. Access the Audit Command Console on all nodes (menu 6). If this is the
first time it is being accessed the Auto Audit field should indicate *OFF. (If
for some reason the console indicates *ON, turn it off using F11. It should
remain off until all audits have run at least once.)
NOTE
The audits displayed will vary, depending on replication options on
your system (e.g., the spool file audit will not display in the list of
active audits if Spool File Replication is not active).
NOTE
Although there are a few exceptions, most audits are scheduled only
on the primary node. Exceptions are noted in the appendix “iTERA
HA Scheduled Audits and Other Jobs” on page 263. Audits that
should not be scheduled on the target are automatically restricted
from being scheduled via the console.
2. Select option 2=Change for an audit.

Managing the Audits
3. Select F7=Create Job.
4. The IBM Add Job Schedule Entry window is displayed. Several fields,
such as Job name, Command to run, Frequency, etc., are pre-populated with
the minimum recommended run settings. Verify that the parameters meet
the run criteria for the audit. While some of these settings may be
changed, do not extend the run frequency beyond the default setting for
the scheduled entry. For example, you may schedule a job to run more
frequently than what has been programmed in the default, but do not
schedule it to run less frequently. Additionally, if scheduled to run more
frequently, ensure it does not run concurrent with other audits, as
described previously.
IMPORTANT
Do not change the name of the job schedule entry. In order to have
the job’s run status properly reported back to the Audit Command
Console, the default job name must be used. Failure to keep the
default name will result in the Audit Command Console initiating
the audit independently of the scheduled job, thus having the audit
run more than is necessary and/or conflicting with other audits.
For the Scheduled time field, enter the time so that the audit will have had
sufficient time to complete prior to the time indicated in the Next Sbm
Time field in the option 2 screen of the console (see the highlighted
example in the Audit Command Console screen, below).

Chapter 8: Auditing
5. Press Enter to add the scheduled entry. The display returns to the Audit
Command Console Change screen. If desired, press F8=View Job to view
the scheduled job in the IBM Work with Job Scheduled Entries screen,
then F3 to return to the console.
6. Repeat step 1 through step 5 for all audits (recommended) or, at

minimum, for the audits listed on page 181. Use appendix “iTERA HA
Scheduled Audits and Other Jobs” on page 263 to help you work through
the scheduling process.
7. Allow all scheduled audit jobs to run via the job scheduler. (Alternatively,
you may manually execute audits using option 8=Submit.) After the audits
have had a chance to run at least once from the job scheduler, enable the

Audit Detail
Audit Command Console using the F11=Start Auto Audit key from the
main Audit Command Console window. The screen below is displayed.
8. Do one of the following:
– For any audits that have not already run, specify a time of day for the
remaining audits to start (e.g., when system resources are more
abundant), then press Enter. Those audits will initiate at that time in
conjunction with the specified audit interval.
– To start the console immediately, press F7 to bypass the time interval
restriction. Unless scheduled in a job scheduler, future audits will be
initiated based solely on the specified audit interval and the Next Audit
Submission date and time field will indicate when each audit will run
in the future.
IMPORTANT
The Auto Audit feature must be active on all nodes. If the Auto
Audit feature is off, then audit job status will not be reported back to
the console and the console will not initiate audit jobs for any audit
that may not have run.
Audit Detail
The following table contains a brief summary of each audit. For additional
information, consult the iTERA HA v6.0 Reference Guide.

Chapter 8: Auditing
NOTE
If an audit below is not displayed in your systems, it is usually
because the component has not been enabled in the Replication
Options screen (30.23).
Audit Name General

Details Results
in Console name
ALT_NAMEP Alternate Name Ensures that alternate names for files are replicated correctly. Alternate names are WRKSPLF on the iTERA
Audit used primarily in SQL applications. The audit submits a job on the primary. outq
When complete, it then submits a job to the target. Discrepancies are
automatically corrected. The command E2AUDALT can also be used to
perform the audit.
A report of the results of the audit lists any corrections that were made. This
report is available on the target and can be accessed using WRKSPLF on the
iTERA outq. To display the alternate names use the command DSPFD. A file
may have more than one alternate name.
ALVL_AUDIT Auditing Level In order to replicate objects correctly, object types must have the correct audit Option 1 displays the
levels. Object Audit Level
The audit reviews all object types and changes the audit level for any objects that Definition screen.
have an object type that is different from our recommended setting. The change
date for the objects on the primary will be modified if the audit level is changed
by the audit.
See Object Audit Level Definitions (4.62) in the iTERA HA Reference Guide
for additional details and a for a list audit level definitions that will be checked
by the audit.
AUDSTREAM Audit Stream AUDSTREAM runs the following audits: To verify that the audits ran,
• CHKOBJMTCH check option 5=History,
• JRNOBJJRN then check for data in the
Record Audit Maintenance
• JRNOBJLST
screen (1.22) on the target.
• RCDCNTCHG (daily, except Saturday) or RCDCNTALL (Saturday) If 1.22 has been cleared
If scheduling the AUDSTREAM, do not schedule these individual audits. since the last time the audits
When AUDSTREAM runs, the status for each of the individual audits is ran and there is now more
updated. data, it is a good indication
If needed, the source for this program can be found in source file that the audits ran as
ITHA/E2.CUST or ITE2/E2.CUST. To customize this program, copy the scheduled.
source member to source file ITHAxx/E2.CUST or ITE2xx/E2.CUST and Because it runs several audits
compile the program into ITHAxx or ITE2xx. (If needed, your Vision Solutions with multiple purposes,
Services Consultant will assist you in modifying the standard programs.) option 1 is not valid for the
Do not allow the AUDSTREAM to run while the DBR_AUDIT or the AUDSTREAM. Option 1
OWNER_AUTH audits are running. can be used for the
individual audits executed
by AUDSTREAM.
IMPORTANT
Special configuration is required for the CHKOBJMTCH audit
in environments with multiple target nodes. Contact
CustomerCare for details.

Audit Detail
Audit Name General

in Console name Details Results
CHKOBJMTCH Check Object Check the iTERA HA

Match IMPORTANT Message Log (E2MSGLOG
This audit is submitted in the AUDSTREAM. or 1.1, F11) on the target
node. Objects added will be
listed in the message log.
Option 1 is not valid for the
This audit submits a job that gathers object information such as creation date, CHKOBJMTCH audit.
format ID, etc., in order to ensure that all objects being replicated on the
primary node exist on the target node. The audit does not check for attribute
differences.
Time variances can indicate issues that need to be resolved. If the object exists
on the primary but doesn’t exist on the target iTERA HA will automatically
resynchronize the object by copying it from the primary system to the target
system.
If the object exists on the target but not on the primary, the audit will delete it
from the target if the data area E2AUDDLT is set to “Y”.
NOTE
The object attribute audit checks for attribute differences.
IMPORTANT
Special configuration is required for the CHKOBJMTCH audit
in environments with multiple target nodes. Contact
CST_AUDP Constraints This audit is submitted on the primary. It identifies missing constraints and There are no results to
automatically adds them to the file on the target and updates the target with any display. However, option 1
missing or different constraints. provides access to the Work
with Constraints screen
(4.24).
DBR_AUDIT Database This audit submits a job which identifies all physical files being mirrored and When option 1 is selected,
Relations their associated logical files and verifies that all database relations are identical to the prompted DSPDBR
those on the primary node. command is displayed.
If the target node does not match the primary node, iTERA HA will Enter the appropriate file,
automatically copy missing files to the target node (no other results of the audit library, and output
will be reported). information.
This audit should run at least weekly on the primary node.
IMPORTANT
Do not allow the DBR_AUDIT audit to run while the
AUDSTREAM or the OWNER_AUTH audits are running.
DEV_AUDP Device Identifies missing configuration objects and attempts to create them on the On the target, select option
Configuration target. If unsuccessful, an entry is added to the configuration error screen. This 1 on DEV_AUDT or 5.4
audit is only available if the replication option is enabled. screen.
DIRE_AUDP Directory Entries Identifies missing directory entries and attempts to create them on the target. If On the target, select option
unsuccessful, then an entry is added to the Directory Entry Errors screen. This 1 on DIRE_AUDT or in
audit is only available if the replication option is enabled. the 5.7 screen.
EXIST_DEVP Device Checks for devices and controllers that exist on one node but not the other. The On the target, select option
Configuration primary purpose is to locate devices and controllers that exist on the target but 1 on EXIST_DEVT.
Existence not on the primary. This audit is only available if the replication option is
enabled.
EXIST_DIRP Directory Entry Checks for directory entries that exist on one node but not the other. The On the target, select option
Existence Audits primary purpose is to locate directory entries that exist on the target but not on 1 on EXIST_DIRT or 5.7,
the primary. This audit is only available if the replication option is enabled. F19.
EXIST_JBSP Job Schedule Checks for job scheduler entries that exist on one node but not the other. The On the target, select option
Existence Audit primary purpose is to locate job scheduler entries that exist on the target but not 1 on EXIST_JBST or 5.5,
on the primary. This audit is only available if the replication option is enabled. F19.

Chapter 8: Auditing
Audit Name General

EXIST_OBJP Object Existence This audit checks all libraries (whether being journaled or not) for objects that On both the primary and
exist on one node but not on the other. target nodes, select option 1
However, the primary purpose of the audit is to review libraries not being for the audit.
replicated to see if any necessary object is missing. There may objects that exist
in libraries that are not replicated but are needed in order for the applications to
run correctly. These may include *JOBDs, *JOBQs, and *SBSDs.
If desired, objects on the primary but not on the target may be copied by using
option 3=Copy from primary. However, this is the equivalent of a non-mirrored
library object sync, so the object will not be monitored for changes.
Additionally, if a filter is active, if the object is copied to the target, it will be
removed when the audit runs. Delete or revise the filter to prevent this from
occurring.
IMPORTANT
The audit, by design, does not consider filters that may be in
place. One reason for this is that objects missing on the target
could be an indication of an incorrectly defined filter.
NOTE
This audit can take an extensive amount of time to run and
evaluate. Because of this, default setting for the audit is disabled
from running on both nodes. After initially running the audit,
you may want to disable (hide) it so that indicators will not be
displayed in the console when the maximum audit interval is
reached. However, results (option 1=Work with App) are only
accessible when the audit is enabled (not hidden).
This audit, by design, is hidden. To enable the audit execute the following steps:
1. From within the Audit Console, view all audits by pressing F8=List All
Audits.
2. On the primary node, select option 2=Change for the EXIST_OBJP audit.
3. Press F18=Override.
4. Enter N for the Place audit on hold parameter.
5. Press Enter twice.
6. Repeat these steps for EXIST_OBJT on the target node.
7. On the primary, exit the screen then reenter the Audit Console. Select
option 8=Submit to run the audit. The job AJ_OBJEX is submitted and
runs in the iTERA HA subsystem.
EXIST_OTQP Output Queue Checks for output queues that exist on one node but not the other. The primary On the target, select option
Existence purpose is to locate output queues that exist on the target but not on the 1 on EXIST_OTQT.
primary. This audit is only available if the replication option is enabled.
EXIST_USRP User Profile Checks for user profiles that exist on one node but not the other. The primary On the target, select option
Existence purpose is to locate user profiles that exist on the target but not on the primary. 1 on USR_EXSTT.
This audit is only available if the replication option is enabled.

Audit Detail
Audit Name General

IFS_AUDIT and Integrated File IFS_AUDIT runs on the primary and executes the E2IFSAUD command. The On the target, select option
IFS_REVIEW System IFS_REVIEW audit on the target node provides access to the IFS Audit results. 1 on IFS_REVIEW.
Select option 1 on the audit to display The IFS Audit History screen. This
screen provides information on how many errors there were on the audit. For
IFS_REVIEW on the target node, select option 1 in conjunction with the
appropriate Fkey which pertains to the failure type for which you want to view
details.
If replicating QDLS, then the first time option 1 and the appropriate Fkey is
selected, leave this field blank. The QDLS paths are automatically built. These
paths establish the necessary links to be able to view the QDLS objects within
the audits. It typically takes longer to compile the audit data when building
these paths.
Once the QDLS path information has been built, a Y will automatically be
placed in the field and subsequent use of option 1 will skip the build of the
QDLS object paths. If you need to rebuild the QDLS path information, change
the Y to N or blank.
If not replicating QDLS, enter a Y to skip the build of QDLS paths. Option 5
will then display the audit details for current, primary, and target values.
This audit is only available if the replication option is enabled.
INSTAL_AUD Installation Setup Tests several settings required for correct iTERA HA configuration, such as Option 1 on each node
Audit system values, data areas, TCP, DDMTCP, and network, attributes. displays the Install Audit
The complete list of tests is documented in the iTERA HA v6.0 Reference Log, where discrepancies
Guide. can be investigated and
resolved.
You must be current with iTERA HA PTFs to run this audit.
JOBSCDE_P Job Scheduler Identifies missing job scheduler entries and attempts to create them on the On the target select option 1
Audit target. If unsuccessful, then an entry is added to the Job Schedule Errors screen. on JOBSCDE_T or in 5.5
JRNOBJJRN Journaled Object Option 1 for the Journal to

Audit IMPORTANT Object audit displays the
This audit is initiated by AUDSTREAM. Local Journal Discrepancies
Audit screen.
Option 1 for the Object to

The JRNOBJJRN audit executes three audits; two on the primary, one on the Journal audit on the target
target. node displays the same
• Journal to Object - Primary This audit compares journals from the iTERA screen that is shown for the
HA object definition file (E2POBJ) with the actual journals and ensures Journal to Object Audit on
that everything the actual journals are supposed to be journaling are, in fact, the primary system but in
being journaled by iTERA HA. This audit audits the primary node only.
When option 1 is executed in the Audit Command Console for this audit, the context of the target
the Local Journal Discrepancies Audit screen is displayed. Each journal that system.
is identified to iTERA HA is displayed on the screen and allows you to see
whether there are differences between the iTERA HA definition and the
Journal Active Object and, if so, how many differences there are. User
journals could potentially contain objects for libraries that aren’t being
mirrored. The options on this screen allow you to drill down to view the
objects in error. The first time you run it you will need to submit the audit
refresh (F18). After results have been returned, the right half of the screen
will then display the audit data. The Object Disc[repancies] field displays
the variance between what iTERA HA has defined and the number of
objects that are currently being journaled by the journal.
• Object to Object - Primary - This audit compares the number of objects
being journaled by each journal on the primary node to the number of
objects being journaled by journals of the same name on the target node.
• Object to Journal - Target - This audit runs after both the Journal to Object
and Object to Object Audits have completed.

Chapter 8: Auditing
Audit Name General

JRNOBJLST Journaled Object The results of the audit can

List IMPORTANT be viewed in the Objects
This audit is initiated by JRNOBJJRN, which is initiated by Requesting Sync screen (1.1,
AUDSTREAM. F6). STRJRN in the Reason
for Sync column. indicates
the audit found the
discrepancy and has started
This audit ensures that every object that should be journaled is. This audit journaling the object.
submits a job on each system that builds a list of all journaled libraries and every As you review the results on
journaled object in them. If objects are not being journaled that should be, a a daily basis, you may see a
resync is requested. Throughout the day, as objects are added and deleted, etc., pattern develop. If the same
the journals are updated but the list remains static and the changes won’t show objects are consistently
in the list until you rerun the audit. resyncing, you should
The submitted job looks at each individual file designated for replication to investigate further to
determine where the object is being journaled then updates the object master determine why. For
file, which is used to verify the integrity of objects journaled in iTERA HA. example, the audit level in
This audit should run daily on both the primary and target nodes. QAUDJRN may be set to
*NONE.
Option 1 on the
JRNOBJLST audit displays
the Mirrored Object
Maintenance screen (4.21).
The Monitoring Checklist
includes instructions on
how to set the filters in order
to determine which objects
should be resynced.
LF_AUDP Logical File Compares the attributes of logical files on the primary system and changes the Option 1 displays the i5/OS
Attributes attributes on the target if they are different. Programming Development
Manager (PDM) screen. For
target node results, use opt 1
on LF_AUDT on the target.
LIB_AUDP Library Compares the library attributes between the primary and target and then change On the target select option 1
Attributes the target library to match the primary if necessary. It is executed on the on LIB_AUDT.
primary.
OBJATRAUD Object Attributes Compares attributes for all objects in mirrored libraries between primary and 1.23 on target.
target. Objects with attribute differences are automatically resynced. This audit
uses the transport journal technology to communicate object information
between source and target systems.
The complete list of attributes audited is documented in the iTERA HA v6.0
Reference Guide.
You must be current with iTERA HA PTFs to run this audit.
OWNER_AUTH Ownership and Results are not reported.

Authorities Audit IMPORTANT
This audit is initiated by AUDSTREAM. Do not allow the
OWNER_AUTH audit to run while the AUDSTREAM or the
DBR_AUDIT audits are running.
Checks to verify that object authority and ownership on the target node is
identical to that on the primary node. If objects on the target node have
differing authority and/or ownership, they are automatically changed to match
the same objects on the primary node.
On the primary, when option 1 is selected for this audit, the prompted
WRKOBJ command is displayed. Enter the appropriate object, library, and
object type information.

Audit Detail
Audit Name General

RCDCNTALL and Record Count On the target, select 1.22

RCTCNDCHG Audit All Objects IMPORTANT screen. Option 1 is not valid
and Changed This audit is initiated by JRNOBJJRN, which is initiated by for this audit.
Objects AUDSTREAM.
These audits tests whether the number of records in all mirrored objects on all
nodes match at the same point in time. If they don’t match, iTERA HA will
re-sync the objects (but only if iTERA HA is able to allocate the object). If
iTERA HA is not able to allocate the object, you must decide whether or not to
resync the object.
RCDCNTALL runs on Saturday only and checks all objects.
RCTCNTCHG runs daily, except Saturday and checks only for objects that
have been changed since the last audit.
SPLF_AUDP Spool File removes extra reports on the target and requests missing reports from the Option 1 on the
primary. This job is submitted on the primary, but results are viewed on the SPLF_AUDT audit on the
target. This audit is only available if the replication option is enabled. target displays the Work
with Output Queues screen,
where you can search for
specific output queues.
SRC_AUDP Source Member This audits checks source member attributes. If source files are changed using Option 1 displays the i5/OS
Attributes PDM and the source type and/or description is changed then the changes are Programming Development
not copied to the target. The reason is that no entry is created in the Manager (PDM) screen. For
QAUDJRN. This is a problem that has been reported to IBM. However, IBM target node results, use opt 1
has indicated a fix for this issue is not forthcoming. Due to this system on SRC_AUDT on the
limitation, it is necessary to compare the source attributes on a regular basis and target.
change the attributes if different.
STOR_PRC_P Stored Stored procedures (functions and procedures) are usually replicated when the Option 1 on
Procedures and associated file is also replicated, provided the program is ILE. If the program is STOR_PRC_T on the
Functions Audit OPM or not attached to a file then the stored procedure will not replicate. target displays the Work
This audit identifies and replicates missing functions and procedures that are with All Spooled Files
found in replicated libraries and deletes stored procedures that exist on the target screen. Results are only
but not on the primary. reported if additions,
deletions or changes to
The command E2AUDSPRC can be used to perform the audit by one or all
stored procedures were
libraries, for functions, procedures, or both, and for a specific function or
performed by the audit.
procedure. The command can be run in two modes *REPORT which will
identify any differences but not fix the differences and the *FIX mode which will
fix the differences.
You must be current with iTERA HA PTFs to use this audit.
SYSV_AUDP System Values You must be current with iTERA HA PTFs to run this audit. Additionally, the Results are accessible from
Replication System Values (*SYSVAL) component must be enabled in the Replication 5.8.
Options screen (30.23).
Some System Values can be replicated to the target node. This audit verifies that
the system values that are defined to be replicated match. Discrepancies are
automatically corrected.
Select option 21 to audit an individual audit or F20 to audit all.
TRG_AUDP Trigger Audit This audit is submitted on the primary. It identifies missing triggers and Option 1 on the audit on
automatically adds them to the file on the target and updates the target with any the primary displays the
missing or different triggers. Work with Triggers (4.23)
screen. Option 1 on the
TRG_AUDT audit on the
target displays the Trigger
Error screen.
USRPRF_AUD User Profile Identifies missing users or attribute differences and corrects the users on the
target.

Chapter 8: Auditing
Audit Name General

V6R1_AUDIT OS Upgrade The i5/OS V6R1 release requires all programs, service programs, and modules See below.
Viability Audit (objects) to be converted before they can be used. Any program, service
program, or module compiled to run on a release prior to V5R1 and where
observability has been removed cannot be converted and therefore, cannot be
used. (This issue is similar to the conversion from CISC to RISC systems.)
Depending on how the system value QFRCCVNRST (*SEC-Force conversion
on restore) is set, the objects will not be converted until used, or they will be
converted when they are restored. If the object cannot be converted and the
system value QFRCCVNRST is set from 2 through 7, then the object will not
restore. There will be only a diagnostic message indicating the object could not
restore but no indication as to why it could not restore.
Many vendor applications and user applications will not convert. It is important
that you start auditing your systems well in advance of the upgrade to make sure
you do not have problems.
This audit should be executed on all systems that may be upgraded to V6R1 of
the OS. The audit should run monthly until all objects that cannot be converted
have been addressed and resolved. This audit does not address any programs
stored in the IFS.
How to Run the V6R1 OS Upgrade Viability Audit

To run the audit, execute the following steps on all systems:
1. Access the Audit Command Console (menu 6).

2. Select option 1 on the V6R1_AUDIT. The following screen is displayed:
3. Press F10 to submit the audit for all libraries. Once the analysis is
complete (or if the Audit Command Console has already run the audit)
press F7 to view the library summary list.
NOTE
Print capability has been provided (F18, F19, F20) in order to be
able to distribute reports to the various individuals who require
them. Reports can also be printed by library from within the
F7=Summary screen.

How to Run the V6R1 OS Upgrade Viability Audit
4. A “1” in the Search for field for library entries with values in the Problem
Objects field displays the libraries that have objects that cannot be
converted. Review and/or print, as necessary.
5. Option 5=Detail on a library displays the conversion details.
Notable information on this screen includes the value in the Can Convert
column. An “N” indicates the object that will not convert. These objects
need to be addressed by the appropriate application developer.
– For user-generated applications, consult with the appropriate

application programmer.
– Third-party applications will most likely require application upgrade.
The First Release column indicates the OS version level at which the
objects were compiled. All objects with observability removed must be
compiled at V5R1 or later.

Chapter 8: Auditing
6. Option 1=Select on a library in the Details view will display detail about
the object. Review objects as necessary.
7. Press F12 twice to return to the Conversion Summary screen, then select
option 7=Errors to view Conversion Errors.

Other Audits
8. Run the V6R1 audit regularly until all problems have been resolved.
IMPORTANT
The Audit Command Console will display an error status for the
audit until all conversion errors have been resolved. Consequently,
the Audit Console test in the Role Swap Readiness Monitor will
display an error status.
Other Audits
The following system checks are not part of the Audit Command Console but
should be executed on a weekly basis and prior to a Role Swap.
User Profile Component Checker (1.50.2 – Both)

This option allows you to audit the primary and target nodes for any objects
used by User Profiles that exist on one node but not on the other. It shows
what components are missing on the current node. However, it must be run on
both nodes. The screen that is accessed displays warnings and errors. Warnings
are informational only and will not prevent the User Profiles from being used
in the event of a Role Swap. Errors indicate critical problems, which will
prevent the User Profiles from being used after a Role Swap and should be
resolved.
Job Description Component Checker (1.50.3 –

Both)
This option allows you to audit the primary and target nodes for any objects
used by job descriptions that exist on one node but not on the other. The
screen that is accessed displays warnings and errors. Warnings are
informational only and will not prevent the job description from being used in
the event of a role swap. However, errors indicate critical problems, which will
prevent the job description from being used unless resolved.

Chapter 8: Auditing

Role Swap
9
Overview
This chapter contains an overview of the four types of role swap available in
iTERA HA, detailing the benefits of and recommended frequency each, and
in what circumstances it is appropriate to use a particular type.
Each time you perform a Role Swap, Virtual Role Swap Test, or Failover,
download the current guide published on Support Central in order to ensure
you have the most up-to-date information available.

Chapter 9: Role Swap
Four Classifications of Role Swap

There are four classifications of role swap within iTERA HA. The following
provides highlights and attributes of each type of test, followed by a full
overview of each test.
Role Swap
Type Attributes
During the test, the primary system stays active, with no impact
on users. Tests are performed on the backup node.
• Testing can be performed during normal work hours.
Virtual Role • Every thing that can be directed to the backup or test system
Swap Test
can be tested.
• Records changed during the test are identified using the
ZZ-audits and reversed using the Heal process (except IFS,
MQ, and Spool Files).
The Virtual Role Swap Test with Communications is executed by

performing two additional steps within the Virtual Role Swap Test
procedure. By testing communications, the test simulates a condi-
tion closer to that which would occur during a Role Swap.
• During the test, the iTERA HA subsystems are ended, the user
Virtual Role subsystems on the primary are ended, and test users are redi-
Swap with rected to the backup node.
Communi- • Communications can be tested.
cations Test
• Applications can be tested on the backup node.
• The primary will not be updated with any changes that have
been made on the backup.
• Records changed on the backup during the test are identified
using the ZZ audits and reversed using the Heal process.
• Users on the primary will be down for the test.
• Communications are tested.

• Applications are validated on the new primary node.
Role Swap
• Users on the primary will be down during the switchover.
• Users will be transferred to the new primary node.
• All processing performed on the new primary is replicated to
the new backup.
• Generally performed because the primary node is unexpectedly

down.
Failover
• It cannot be run if the iTERA IP address on the primary is
active.
The following sections provide additional detail about each type of role swap.

Four Classifications of Role Swap
Virtual Role Swap Test

The Virtual Role Swap Test provides the ability to test applications on the
target system and can be executed at any time with minimal impact on the
production system.
During the test, the apply and syncing processes are temporarily suspended on
the target node, even though iTERA HA continues to monitor these processes
on the primary. The updates from the primary continue to accumulate in the
remote journals on the target. The iTERA HA menus on the backup system
indicate you are in virtual mode. Users can execute applications on the backup
node in order to test functionality. Records can be added and deleted from
files. Meanwhile, iTERA HA keeps track of changes made to the systems. The
changes which occur in the virtual environment are not sent to the primary.
While application testing occurs, heal records from the primary are requested
for any objects being changed on the target so that they are in the remote
journal on the backup node and ready to be applied when the virtual test is
ended.
IMPORTANT
Be aware that if files are deleted or members are cleared on the target,
a resync of the deleted/cleared data will occur.
NOTE
Heal is not supported for WebSphere MQ, IFS, and Spool Files.
Additionally, the audits are not currently equipped to identify
changes made to these components during the Virtual Role Swap.
Therefore, these components are not supported in the Virtual Role
Swap and should not be tested. However, if a complete resync of IFS
is practical, it can be planned as part of the process.
IMPORTANT
Keep in mind that since all processing and changes performed in a
Virtual Role Swap will have to be reversed, you must allow sufficient
time for the changes to be rolled back, which takes time. We
recommend you only test a few things with your first one in order to
gauge how much time it will take your system to recover, then
increase the amount of processing with each subsequent test so that
you don’t encounter an unacceptable reversal time.
To execute a Virtual Role Swap, follow the procedure, see “Virtual Role Swap
Test and Virtual Role Swap with Communications Test” on page 222 of this
guide.

Virtual Role Swap with Communications Test

The Virtual Role Swap with Communications Test incorporates the best
features of both the Virtual Role Swap Test and the Role Swap without the fear
of inadvertently updating your production system. Changes made on the
backup are not sent to the primary and are reversed on the backup during
recovery phase. However, all normal processing must be ended on the
production system (users cannot be on the primary system) during the test and
the same limitations exist as in the regular Virtual Role Swap Test, as described
above. The same basic procedure is used with additional steps included for
disabling and enabling communications.
To execute a Virtual Role Swap with Communications Test, follow the

procedure see “Virtual Role Swap Test and Virtual Role Swap with
Communications Test” on page 222.
Role Swap
The role swap process can be executed in order to have the backup node
assume the role of a fully-functional primary node. During the switchover
process, users are redirected to the new primary and normal production is
resumed. When ready to do so, another role swap is performed to return to the
original production node. Some customers prefer to run for only a short period
on their new primary machine and switch back quickly, while others prefer to
run in a swapped condition for an extended period of time.
In our experience, the actual role swap process usually takes less than forty-five
minutes to complete, so user-downtime is greatly minimized.
Prior to performing a role swap, audits and other system checks help ensure the
viability of the backup node assuming the role of the primary node.
The purpose of a role swap is to:
1. Verify correct replication and that everything that is needed to run the
system from the new primary is in place and ready to be used.
2. In the event of a failover, a role swap will provide good idea of what is
required to be fully operational and how long it will take to restore (A
failover will actually happen faster than a role swap, since in a failover
certain assumptions are made that are not made in a normal role swap.)
3. If you need to take your production node down for maintenance, the
backup assumes the role of a fully-functional primary node, minimizing
user downtime.

Preparation and Procedures
NOTE
A replicate node cannot be used in a role swap configuration. If more
than one backup node is defined, the role swap is performed to the
node identified as BACKUP 1. In order to roll to the other backup
node it must first be promoted to BACKUP 1.
To execute a role swap, see “Role Swap Procedure” on page 205 of this guide.
NOTE
Because each environment is unique, you may need to document
additional role swap procedures unique to your environment.
Failover
IMPORTANT
A failover should only be executed in the event of failure of the
primary system.
IMPORTANT
If you have experienced an actual failure on your primary node call
Vision Solutions CustomerCare immediately. As part of your
support agreement, Vision Solutions will help you work through a
failover situation.
To execute a failover, follow the procedure detailed in the Failover Procedure

chapter of the iTERA HA v6.0 Advanced Features Guide.
IMPORTANT
By providing the failover procedure, Vision Solutions does not imply,
recommend, nor suggest that you perform a failover on your own.
Preparation and Procedures

Thorough preparation is vital in ensuring a smooth role swap (or virtual role
swap, etc.) procedure. There are a variety of required pre-check processes you
must execute in iTERA HA that will let you know whether your nodes are
ready, and what needs to be corrected in order to make them ready. These
include performing the tasks on the iTERA HA Monitoring Checklist,
including all audits in the Audit Command Console and performing all tests in
the Role Swap Readiness Monitor.

To guide you through each role swap process from start to finish, detailed
instructions are included in this guide.
IMPORTANT
Prior to performing any type of role swap, always download a new
copy of this guide from Support Central to ensure you have the latest
version.
How to Determine the Type of Role Swap to Perform

The following table indicates the tests that can be performed and
recommended frequency for each type:
Virtual Role
Virtual Role Swap with Live Role Failover
Swap Communications Swap
Test
Recommended Frequency Monthly Optional Quarterly Only if

necessary
Tests:
Green Screen Applications Yes Yes Yes Yes
License Keys Yes Yes Yes Yes
Data Integrity Yes Yes Yes Yes
Data Input Yes Yes Yes Yes
Data Query Yes Yes Yes Yes
MQSeries No Yes Yes Yes
Web-based Applications No Yes Yes Yes
FTP in/out No Yes Yes Yes
EDI Packages No Yes Yes Yes
Interfaces No Yes Yes Yes
iTERA Role Swap Process No No Yes Yes
Updates from the Production N/A N/A Yes Yes

replicated to Target

Role Swap Configuration Examples
IMPORTANT
The Virtual Role Swap Test and Virtual Role Swap with
Communications Test can help prepare your systems for live role
swap and/or failover events. These tests do not replace the need to
perform a live role swap regularly, but can help you to identify many
potential issues that would prevent a role swap or failover from being
executed successfully.
Role Swap Configuration Examples

While there are many possible system configurations, following are two
examples of possible configurations.
Two-node CRG
Role swap configuration for a standard two-node CRG is depicted in the
following example. The systems can be co-located or in separate data centers.
When the role swap is executed in an environment with a standard

source-to-target replication configuration, the target assumes the role of a
fully-functioning source node and the former source node becomes the target
node.

Three-node CRG
A role swap configuration for a three-node CRG consisting of a source and
target node located in the same data center and a target node located in a
separate off-site location is depicted in the following diagram.
The normal production mode in this example shows the source (upper left)
replicating to target 1 (lower left) and target 2 (upper right).
In high availability mode, a role swap has occurred where target 1 is now the
source and is replicating both to the former source (now target 1) and target 2
off-site.
In maintenance or disaster recovery mode, target 2, located off-site, becomes

the source and the systems located in data center 1 become the target nodes
when replication processes are resumed.

Role Swap Procedure
Role Swap Procedure

This procedure is used for performing a role swap on your production CRG. If
you need assistance, contact CustomerCare at 800-337-8214 or
949-724-5465.
IMPORTANT
Vision Solutions recommends that you review this complete
document a week prior to performing a role swap in order to refresh
your knowledge, better understand the process, and ensure that you
are aware of, and account for, the unique equipment and processes
within your environment.
IMPORTANT
Document separately any additional steps that you must perform for
your particular environment.
Vision Solutions recommends that a role swap be performed, at minimum, on

a quarterly basis and the virtual role swap test monthly (or whenever major
changes to applications occur).
When preparing to perform your first role swap, Vision recommends that you
schedule a System Health Check through our Professional Services department
(this is a billable service). While optional, the System Health Check will help
ensure that iTERA HA is correctly configured and is ready for the role swap.
The System Health Check includes additional role swap training, if needed.
NOTE
If performing a migration, extensive testing of your backup node is
required. Consult with the Professional Services department for
pricing and additional details.
A role swap can only be performed on a machine defined as BACKUP 1. In

order to perform a role swap to BACKUP 2, it must first be promoted to
BACKUP 1. See “How to Promote Backup 2 to Backup 1” on page 233 for
instructions. A REPLICATE node cannot be used in a role swap.

Section One: Role Swap Pre-Planning Procedure

IMPORTANT
This section should be performed at least one week prior to the role
swap.
Assi-
Section One: Role Swap Pre-Planning Procedure gned Done
1. Ensure the Monitoring Checklist is run regularly each day up through the day of the
role swap. Correct any issues that exist. Do the steps in the weekly section 24 to 48
hours before the role swap.
2. Ensure audits in the Audit Command Console are running regularly and that all
audits have been run at least 24 hours prior to the role swap. Correct any issues
before proceeding with the role swap.
3. Review the list of all non-IP hardware and update the plan for switching them during
the role swap. If not already done, develop a written plan for handling all non-IP
devices during the role swap. Otherwise, review the list and verify that the actions are
still correct.
4. Review Device Configuration replication to verify that all necessary devices have
been replicated. This could include remote printers, handheld scanners, or other
devices that require special device configuration. Review the device configuration
replication screen (5.4).
5. Verify that you have an alternate sign on method and test it prior to performing the
role swap. Vision strongly recommends that a group of display devices be attached to
QCTL or *BASE. Remember that iTERA HA will be ending the takeover IP address
and you may be ending your version of QINTER. If devices have not been defined to
QCTL or *BASE the console may be used to sign on after the role swap.
6. Using option 30.22 on all systems, review all IP addresses. List what they are used for
and how they will be handled during a role swap. This is especially important for all
user IP addresses listed for the Primary and Backup 1 nodes.
NOTE
If you do not want your Takeover IP address to end on the primary you must
change the IP Mapping in 30.22 on the primary.

Role Swap Procedure
Assi-
Section One: Role Swap Pre-Planning Procedure gned
Done
7. Document the processes you will use to end and start user jobs. Vision recommends
that you update the E2USRDWN and E2USRUP programs to perform these
functions for you. Copy the E2.CUST file from ITHA (or ITE2) to ITHAxx (or
ITE2xx). Modify and recompile the E2USRDWN and/or E2USRUP CL programs
in the CRG library. E2USRDWN and/or E2USRUP may be different between
nodes.
NOTE
If you are replicating WebSphere MQ, include in your plan the processes for
ending the MQ Queue Managers and starting the MQ Subsystem.
8. Develop and document your test plan. Indicate all testers and what they will be
testing. Review and revise it, as needed.
9. Use the command WRKACTJOB to review and document the user jobs and
processes running on the primary system. Print the screens for post-role swap
comparison. After the role swap process, you will be checking the subsystems on the
new primary to ensure that the requisite jobs are running.
10. Ensure that the iTERA HA software is on the current PTF release level. If not
current, make sure all iTERA HA and Cross Product PTFs have been loaded and
applied the week before the role swap. (Also load and apply iTERA Alert PTFs, if
using iTERA Alert.)
IMPORTANT
Do not load PTF updates during the week of the role swap (unless you are told
by CustomerCare that you need to fix a specific issue).
11. Review the preferred states for all iTERA-managed triggers on all nodes (4.23).
Generally, triggers should have the following:
– Preferred State - PRI = Enabled

– Preferred State - BKP = Disabled
Refer to the iTERA HA v6.0 Reference Guide for more information.

12. Review the preferred states of all iTERA-managed constraints on all nodes (4.24).
Generally, iTERA-managed constraints should have the following:
– Preferred State - PRI = Enabled

– Preferred State - BKP = Disabled
Refer to the iTERA HA v6.0 Reference Guide for more information.

Assi-
Section One: Role Swap Pre-Planning Procedure gned
Done
13. Check the iTERA HA subsystem on all nodes to ensure they will run enough jobs.
The default number of jobs in the iTERA subsystem is 100 (50 for the E2SYSJOBQ
and 50 for E2JOBQ). A large number of journals will increase the need for
additional jobs.
Verify that the Maximum Active Jobs on the primary is set the same as (or higher
than) on the current backup.
a. Execute the following command on both the primary and backup 1:
WRKSBSD E2xxSBS
(where xx is your CRG code)
b. Select option 5=Display

c. Select option 1=Operational attributes.
d. Compare and verify that the Maximum jobs in subsystem field on the primary is
equal to or larger than all other nodes. If not, return to the Work with Subsystem
Descriptions screen (F12 twice), select option 2=Change, and change the value for
Maximum jobs, then press Enter.
e. Select option 5=Display from the Work with Subsystem Descriptions screen.
f. Select option 6 to display the job queue entries.
g. The total number of entries for both job queues should equal the number
specified in the subsystem description. To change the number of job queue
entries, use the following commands.
CHGJOBQE SBSD(E2xxSBS) JOBQ(E2SYSJOBQ) MAXACT(nnn)
CHGJOBQE SBSD(E2xxSBS) JOBQ(E2JOBQ) MAXACT(nnn)
(where xx is your CRG code and nnn is the number of entries)

14. Clear files on the current primary using the following commands:
CLRPFM E2P5101C
CLRPFM E2POSR
CLRPFM E2PZZLG
The E2P5101C file contains the heal records. The file exists on all nodes but is only
used on target nodes.

Role Swap Procedure
Section Two: Pre-Role Swap Checks

Perform the steps in this section within two hours of the role swap.
IMPORTANT
If you have more than one CRG verify that you are signed on to the
correct CRG on all nodes. You can check this by looking at the
iTERA HA Menu (top row, second from the left). Also verify that
you are signed on with the iTERA HA Admin profile.
Assi-
Section Two: Pre-Role Swap Checks gned Done
1. On all nodes, review scheduled jobs. Place any scheduled jobs that may submit
during the role swap on hold.
• To hold replicated jobs, use 5.5, F19 on the primary. If you don’t use 5.5 to hold
the jobs (e.g., if you hold them using another method), note which jobs you put
on hold so that you can release them after the role swap (in “Section Eight: Final
Steps”).
IMPORTANT
Do not use 5.5 to hold the jobs on systems where there is more than one
backup node or if there is a replicate node defined.
• Hold the Journal Manager job XPJRNMGT (3.33; all nodes).

2. End then restart the iTERA HA subsystem jobs.
• E2ENDSBS - all nodes; primary first, then targets

• E2STRSBS - all nodes; targets first, then primary).
The purpose of this step is to recycle the job logs so that when the actual role swap is
executed, the subsystems will end more quickly.
3. End the Audit Command Console on all nodes (1.8, F11=End Auto Audit).
4. If this is a migration you may want to save the local journal receivers on the current
primary into a save file.
5. In Non-Mirrored Object Replication Check (4.30, primary), verify that all objects
that need to be replicated are being replicated. The Last Sync Date/Time data
should be current. If not, use F9=Sync NM-Library to sync all objects. (This should
be scheduled to run each day.)
6. If you have a multiple target node environment, use menu 30.7 on the node that
will become the new primary. Test communication to all other target nodes in the
CRG. Correct any issues prior to continuing.

Section Three: Final Pre-Role Swap Checks (30

minutes before role swap)
Perform the steps in this procedure 30 minutes prior to the role swap.
Assi-
Section Three: Final Pre-Role Swap Checks (30 Minutes Prior) gned Done
1. Optional: Notify users that the system will be coming down in thirty minutes.
2. Perform the following steps to verify that the system is ready for the role swap. The
items listed below should be checked in the order listed. Any issues encountered
should be investigated and resolved before proceeding.
a. Update the System Monitor (1.1 F10, primary). Check the following:
– Verify that the % Total Disk Storage Used displayed for the Backup 1 node
well below the acceptable threshold.
– Local/Remote Journals Active, Apply Jobs Active and Network/Subsystem Active
should all indicate YES values (there should be no *NO indicators).
– Journal Entries Not Applied should either be caught up or a relatively low
number of entries.
– Objects Requesting Sync should be NONE.
– Press F14=Role Swap Readiness.
– The AUDIT_MON test may be in WRN status because the Audit
Command Console has been ended.
– The JOBSCDE test may be in ERR status because the journal manager
job has been placed on hold.
– No other tests should be in WRN or ERR statuses.
b. Review the Record Count Audit results (1.22; backup).
c. Check the Heal Status Summary screen (3.7; backup). Verify there are no
pending Heal requests.
d. On all nodes, review the iTERA HA message log (1.1 F11=E2MSGLOG).
Ensure there are no high severity errors occurring. If so, resolve them. Do not
proceed with the role swap until issues are fully resolved.
e. On all nodes, check the iTERA HA subsystem to ensure it is active and that
there are no jobs in MSGW status (1.1, F7=E2SBS).
3. Place the Role Swap Readiness Monitor job on hold on all nodes (E2SBS, opt 3 on
xx_RSRMON).

Role Swap Procedure
Section Four: User Shutdown

NOTE
In all subsequent sections of this checklist, QRB stands for Quick
Rollback and is used to indicate the minimum steps that must be
performed if returning to the original primary after only a short
period of limited testing. Perform all subsequent role swap steps if
you allow your users or IT staff to update more than a limited
amount of easily verifiable data.
IMPORTANT
Verify all users are finished processing before ending the subsystems.
IMPORTANT
During the actual role swap, never go into a restricted state which
will end all subsystems and sever communications.
All steps in this section are executed on the primary node.
Assi-
Section Four: User Shutdown gned Done
1. End all user applications and user-related subsystems. (QRB)

a. If you have updated the E2USRDWN exit program use option 40.20 to end all
user jobs. Otherwise, use 40.21 (or WRKACTJOB) to end all third-party
applications. End your version of QINTER to ensure that users no longer have
access to the primary.
b. End all user interfaces (e.g., ODBC and JDBC connections that do not use the
User Takeover IP address).
2. If you are using WebSphere MQ Replication, end the Queue Managers, verify that
the MQ apply jobs are caught up, then end MQ Replication (QRB).
a. End all MQ Queue Managers using your company’s established protocols. Make
sure the QMQM subsystem is not active.
b. Verify that the MQ mirroring process is caught up using the Mirror Process
Monitor (1.1, F16=Process Monitor). Verify that there are no exposed entries for
the MQ journals.
c. End iTERA HA WebSphere MQ Replication (5.6, F17=End MQ replication).

Assi-
Section Four: User Shutdown gned
Done
3. Sync exclusively locked objects.
a. Select 40.24.
b. Do one of the following:
– To replicate individual objects in the list, select option 9=Sync Object for any
necessary objects
– To replicate all objects in the list, press F10=Sync Roll Request. Press F10 to
confirm replication of all necessary exclusive locked objects.
4. Check Data Queues (required for V5R3, optional for all other OS versions)
Select 40.23. If the message “All data queues are empty” is displayed at the bottom of
the screen then continue with the next step. If the message indicates there is data in
any data queue, review the message log to see which data queue needs to be emptied
(E2MSGLOG – primary). If there are entries in any data queues, determine why. If
applications are still processing them, they must be caught up before executing the
role swap.
5. Verify that there are no other jobs running. (QRB)
a. Select 40.22, Monitor New Journal Entries

b. Press F7=Reset Start.
c. Periodically press F5=Refresh and check to see if the number in the New Entries
column changes (press Page Down, if necessary). If it does, there are jobs running
that must be ended. Use option 7=Display New Entries to determine which jobs
are still running. Decide whether to the cancel the job or allow it to finish
running.
6. Check the Object Monitor (1.4) to verify that it is caught up. (QRB)
• The Present Delay field should be less than 10 or so.
• IFS remain to process should be 0 before continuing.
7. If replicating spool files, check the Spool File Monitor (1.6). Verify the To Be Processed
value is zero.

Role Swap Procedure
Assi-
Section Four: User Shutdown gned
Done
8. Perform final check of the System Monitor. (QRB)
Select 40.25, F10=Update Monitor. Verify there is no latency, that there are no
objects requesting sync, and that there are all ‘YES’ indicators.
NOTE
If you replicate WebSphere MQ, the apply jobs and journal status will indicate
*NO.
9. Select 40.22 then press F5=Refresh to verify again that there are no new entries
caused by user jobs (entries from QAUDJRN are acceptable). (QRB)
IMPORTANT
Do not reset!
If there are new entries, it indicates there are still jobs running. Use option 7=Display
New Entries to determine which jobs are still running. If any new entries are from
user jobs, decide whether to end the jobs or allow them to finish processing. User jobs
must be completed prior to executing the role swap.

Section Five: Perform the Role Swap

IMPORTANT
Make certain that you are signed on with the iTERA HA admin
profile and that you are the only person signed on with it. Two ways
to determine this are as follows:
-Use WRKACTJOB and locate the subsystem that is used for

interactive jobs.
-Use the command WRKSBSJOB SBS(sbsname)

(Replace “sbsname” with the subsystem your system uses for
interactive jobs). For example, WRKSBSJOB SBS(QINTER).
IMPORTANT
If you attempt a role swap then need to abort it for any reason,
contact CustomerCare immediately for assistance.
IMPORTANT
NEVER end the apply jobs from this point on.
Assi-
Section Five: Perform the Role Swap gned
Done
1. Execute the role swap on Backup 1. (QRB)
a. Select menu option 40.30.

b. Select F16=Role Maintenance.
c. Select F16=Initiate Roleswap.
Verify that the screen heading indicates “Initiate Role Swap Confirmation”.
2. Review confirmation screen. (QRB)
IMPORTANT
Verify that the screen indicates “Role Swap”. If it indicates “Failover”, do not
continue with the role swap. Most likely, it indicates there is a communications
failure between systems. Contact CustomerCare immediately in order to
troubleshoot and resolve the issue.

Role Swap Procedure
Assi-
Section Five: Perform the Role Swap gned
Done
Review the parameters and set as needed, then press F10 to accept.
NOTE
Vision Solutions recommends setting all parameters on the confirmation
screen to *NO for the first role swap. For subsequent role swaps, set as desired.
3. Check Role Swap Progress
IMPORTANT
Do not change any iTERA HA files or data areas, end any iTERA HA
programs, or start the iTERA HA subsystem manually.
Monitor the role swap from the Role Swap Monitor screen on the backup node
until the role swap has completed. The Status field in the middle section will
indicate “Complete” when the role swap has finished.

Section Six: Post Role Swap Activities
Assi-
Section Six: Post Role Swap Activities gned Done
1. Sign off all nodes, then sign back on to all nodes with the HA admin profile. (This
clears the cache memory and QTEMP.)
2. Review the System Monitor (QRB)
a. 1.1, F10 – primary. Verify that the journals and apply jobs are active. (YES status
in all appropriate columns.) (You may need to update the monitor more than
once if all statuses are not correctly reported.)
b. Press F7=E2SBS. Verify that the subsystem and all necessary jobs are active.
c. Press F14=Role Swap Readiness. Verify that no tests in the Role Swap Readiness
Monitor (with the possible exception of IPCONWRN) are in ERR status.
NOTE
If you are replicating WebSphere MQ, the apply jobs and journal status will
indicate *NO. The Role Swap Readiness Monitor will also show in error.
3. Check Triggers (4.23) on all nodes (QRB)
Verify that the triggers are not in an error status. An error status is indicated by the
Trigger State column showing red text. (If the library name shows in red text, it
indicates the library is no longer being replicated and you do not need to correct the
triggers in that library.) You may need to set other triggers to the node’s preferred
state. For more information on triggers, refer to the iTERA HA v6.0 Reference
Guide.
NOTE
iTERA HA does not modify the state of triggers in non-replicated libraries.

Role Swap Procedure
Assi-
Section Six: Post Role Swap Activities gned
Done
4. Check Constraints (4.24) on all nodes (QRB)
Verify that the constraints are not in an error status. An error status is indicated by
the State column showing red text. (If the library name shows in red text, it indicates
the library is no longer being replicated and you do not need to correct the
constraints in that library.) You may need to set other constraints to the node’s
preferred state.
Constraints that indicate REQ in both the Pri and Bkp fields (under Preferred State
column) cannot be changed. For more information on constraints, refer to the
iTERA HA v6.0 Reference Guide.
NOTE
iTERA HA does not modify the state of constraints in non-replicated libraries.
5. If replicating spool files, release output queues by running the following command:
E2RLSOUTQ
NOTE
This will release all replicated output queues.
6. If necessary, change the system name (QRB). See instructions in Appendix D,

“i5/OS Recommendations” of the iTERA HA v6.0 User Guide.

Assi-
Section Six: Post Role Swap Activities gned
Done
7. If using ODBC, JDBC, or any other communication protocol, you may need to
change the new primary *LOCAL RDB entry to be the same as it was on the old
primary. See instructions in Appendix D, “i5/OS Recommendations” of the iTERA
HA v6.0 User Guide.
8. If you are using WebSphere MQ replication, start the process. (QRB)
a. On the new primary:

1. Remove WebSphere MQ markers using the command E2DLTMQCPT.
2. Start the QMQM subsystem and all MQ Queue Managers using your
company's established protocol.
3. Check the 5.6 screen and verify there is an assigned default journal (upper-left
section of the screen).
NOTE
If WebSphere MQ Replication has never been set up on this system, you may
need to assign a default journal using F13 and opt 1, assign the journal to a
queue manager, and start journaling.
4. Start WebSphere MQ replication (5.6, F15).

b. On the backup, select Work with Apply Jobs (3.4) and verify the MQ apply jobs
have started. If they have not started:
1. End WebSphere MQ Replication (5.6, F17).
2. Start WebSphere MQ Replication (5.6, F15).

Role Swap Procedure
Section Seven: User Signon

The steps in this section prepare the system for access by users.
Assi-
Section Seven: User Signon gned
Done
1. Move any necessary devices. (QRB)

2. Redirect your user IP address(es) to the new primary. (QRB)
You can redirect the user IP anytime after confirmation screen has been reviewed.
3. Start necessary user subsystems and interfaces, etc. (QRB)
Optional: Call the E2USRUP program, if created. (The CL source program in the
source file ITHAxx/E2.CUST or ITE2xx/E2.CUST.)
4. Optional: If you specified *NO for the Enable Frozen Profiles parameter in the
confirmation screen, enable them now (5.1, F17 – new primary).
5. Notify users that they can use the system (QRB).

Section Eight: Final Steps

IMPORTANT
Do not perform the following steps if you are rolling over from the
original primary to the original backup, and plan on running for
only a short period of time.
Assi-
Section Eight: Final Steps gned Done
1. On all nodes, select option 6 to enter the Audit Console, press F11=Start Auto
Audit, then F7=Ignore Setup Time, to start the Audit Console.
2. Set the job schedule entries. Some environments generate duplicate entries. After
reviewing your environment, if no duplicates are detected, set the job schedule
entries as follows:
a. Set scheduled jobs on the on the new primary:

1. 3.33, option 6.
2. 5.5, F19=Set Status, option P=Use Primary System Status
NOTE
This option does not work if you have a Replicate node defined to the CRG.
b. Set scheduled jobs on the new backup system:

1. 3.33, option 6
2. 5.5, F19=Set Status, option B=Use Backup System Status
If there are duplicate job schedule entries manually hold the entries on the backup
then release only one of the duplicates on primary node using the following
procedure:
a. Select 5.5, F8=WRKJOBSCDE on the current backup. Review the status

column and locate all scheduled jobs. Print screens or otherwise note the
scheduled jobs.
b. Select option 3=Hold on all scheduled jobs.
IMPORTANT
If the jobs are not held on the target machine and they run, then any files
affected by those jobs will be out of sync.
c. On the current primary, select 5.5, F8=WRKJOBSCDE.

d. Using the screen shots of the scheduled jobs that were on the backup, change the
scheduled jobs using option 6=Release on only one of each pair of duplicates.

Section Eight: Final Steps
Assi-
Section Eight: Final Steps gned
Done
3. Clean up the Journal Manager on all nodes.

E2JRMSET
4. Review local journals in use on the new primary.
a. Select 3.32 on the new primary.

b. For each local journal, select option 8=Display Receiver Status.
c. If the In Use column displays “YES”, select F9=Applications, then select option
4=Delete.
d. Press Enter.
e. Press F12 to exit. The In Use column should indicate “NO”.

Virtual Role Swap Test and Virtual Role Swap with

Communications Test
This procedure is used for performing the Virtual Role Swap Test and the
Virtual Role Swap with Communications Test.
• The Virtual Role Swap Test does not affect users on the production
system. However, only limited communications testing can be performed.
• The Virtual Role Swap with Communications Test allows you to perform
all testing available in the Virtual Role Swap, as well as communications.
However, you will be required to restrict the production system so that
users cannot access it during the test.
NOTE
The Virtual Role Swap with Communications Test steps are listed as
optional steps within the Virtual Role Swap Test instructions.
A Virtual Role Swap can only be performed on a machine defined as BACKUP

1. In order to perform the test on BACKUP 2, it must be promoted to
BACKUP 1. See “How to Promote Backup 2 to Backup 1” on page 233 for
instructions. A REPLICATE node cannot be used in a Virtual Role Swap test.
• Execute the Virtual Role Swap procedure on BACKUP 1. See “Execute the
Virtual Role Swap Test” on page 224.
• After testing is complete, end the Virtual Role Swap Test as documented in
the procedure and resume replication. See “End Virtual Role Swap;
Resume Replication” on page 230.
• After recovery from the Virtual Role Swap is complete, DO NOT

promote Backup 1 to Backup 2 until verifying that the ZZ audit jobs have
completed processing the local receivers on the backup node. Refer to
“How to Check the Progress of the Virtual Role Swap Recovery” on
page 234.
Every time you perform a Virtual Role Swap Test, read through the procedure
completely the week prior. This will refresh your knowledge and help ensure
that you are aware of, and account for, the unique equipment and processes
within your environment and will help you better understand the Virtual Role
Swap process. It may also benefit you to review “Virtual Role Swap Test” on
page 199.

Virtual Role Swap Test and Virtual Role Swap with Communications Test
IMPORTANT
Keep in mind that since all processing and changes performed in a
Virtual Role Swap will have to be reversed, you must allow sufficient
time for the changes to be rolled back, which takes time. We
recommend you only test a few things with your first one in order to
gauge how much time it will take your system to recover, then
increase the amount of processing with each subsequent test so that
you don’t encounter an unacceptable reversal time.
NOTE
Heal is not supported for WebSphere MQ, IFS, and Spool Files.
Additionally, the audits are not currently equipped to identify
changes made to these components during the Virtual Role Swap.
Therefore, these components are not supported in the Virtual Role
Swap and should not be tested. However, if a complete resync of IFS
is practical, it can be planned as part of the process.
IMPORTANT

Execute the Virtual Role Swap Test

Virtual Role Swap Test Procedure Pri Bkp
1. Ensure all nodes are in sync. ❏ ❏

• Run all daily and weekly iTERA HA audits within 24 hours of the test.
• Perform the steps on the iTERA HA Monitoring Checklist within 24 hours of the
test.
Optional: If you do not run the Library Analyzer weekly as part of the monitoring
checklist then run it on the primary using 4.11, F18=Submit Library Analyzer.
Verify that all eligible libraries that should be replicated are being replicated.
NOTE
When the Virtual Role Swap Test is executed, any objects listed in option 3.7 on
the backup node (Heal Summary; Pending entries), objects requesting sync, and
objects with record count errors may contain incorrect data during testing. The
data in these objects on the backup may be different than on the primary.
2. Know how you are going to have your users access your backup node. ❏ ❏
• If you are doing a Virtual Role Swap Test with Communications Test you will be
redirecting your primary’s user IP address to the backup node. There are many
ways to accomplish this. For example, DNS server, virtual IP address, intelligent
routers, etc.
• If you are executing the standard Virtual Role Swap Test then you will need to
provide a way for the users who will be performing testing to access the backup
node.
3. Verify that the ZZ-Audits are running. On the backup node enter E2SBS. There ❏
should be one ZZ job for each journal, excluding IFS and Transport journals.
IMPORTANT
ZZ-Audits are required in order to perform the Virtual Role Swap Test. If they
are inactive, objects changed during the Virtual Test will not be healed.
4. If user journals are defined in iTERA HA, verify the setting for the parameter ❏ ❏
FIXLENDTA. iTERA HA requires this parameter to be set to *JOBUSRPGM.
5. Turn off Audit Command Console on all nodes using menu 6, F11. ❏ ❏
6. Place the Role Swap Readiness Monitor job on hold on all nodes (E2SBS, opt 3 on ❏ ❏
xx_RSRMON).

7. On the backup node, hold all jobs in the Job Scheduler (including the Journal ❏
Manager job that would normally run on the backup system) that are scheduled to
run during the time frame of the Virtual Role Swap Test.
NOTE
If you are using Job Scheduler Replication you can change the status using that
process (5.5, F20, H=Hold all jobs), (this will not work if you have a replicate
node). However, if you are going to use the Job Scheduler Replication to hold
and release the jobs on the job scheduler, verify that the Status fields in 5.5 on
the primary (Cur, Pri, and Bkp) are set correctly and understand that the statuses
for all job schedule entries will be changed.
8. On the primary node, only hold the iTERA HA jobs (including the iTERA HA audit ❏
jobs, and the journal manager job) on the Job Scheduler that will run during the time
frame of the Virtual Role Swap Test.
9. Optional: Replicate exclusively allocated objects. Select Work with Locked Objects ❏
(4.22), enter option 9=Sync Object on all records, press enter, and then select
F10=Sync Roll Request to replicate those objects to the backup system.
NOTE
This may not be an option on a Virtual Role Swap Test because users may have
the objects locked.
10. On the primary node, sync non-mirrored objects. (4.30, F9=Sync NM-Library, then ❏
enter the name of the library and target system.) (Note: you can use *ALL for the
library name to replicate all definitions at once.)
NOTE
This should be scheduled to run each day.
11. On primary node, refresh the System Monitor (1.1, F10=Update). Verify that there ❏
are no issues to resolve.

12. If replicating WebSphere MQ, end replication on the primary using 5.6, F17=End ❏
MQ Replication.
NOTE
WebSphere MQ Replication cannot be active during the Virtual Role Swap Test.
IMPORTANT
Do not end the MQ managers on the primary.
13. Review Triggers on all nodes. On all nodes run the Triggers update (4.23, F18=Submit ❏ ❏
Info Build, F8=All Research new Triggers), then review the current status of all the
triggers. Ensure that you understand why they are in their current state. Change the
preferred state, if needed.
14. Review Constraints on all nodes. On all nodes run the Constraints update (4.24, ❏ ❏
F18=Submit Info Build), then review the current status of all the Constraints. Ensure
you understand why they are in their current state. Change the preferred state, if
needed.

15. Execute the Virtual Role Swap Test. From the backup node select the Role Swap ❏
option 40.30, F16=Role Maintenance, F19=Virtual Test.
IMPORTANT
Review the Confirmation Screen. Verify that the screen heading indicates Virtual
Test Role Swap, as indicated:
• For the Process User Startup Program parameter, enter Y if you want to run the
iTERA HA Role Swap Startup program in the “Step 1” option. Otherwise, enter
N and press F10=Initiate Virtual Test Setup.
• The Role Swap Monitor will be displayed and processing will begin. In the Process
section (the center section of the screen), when step 9, Virtual Test Ready, is
displayed with the Status of Complete, you may exit the screen using F3.
NOTE
If it appears to take longer than expected for the completion message to be
displayed, contact CustomerCare.

16. Once you have exited the Role Swap Monitor screen, the message “WARNING: ❏
Virtual Test Role Swap in Progress” will be displayed at the top of the iTERA HA menu
screens. Triggers and constraints have been reset, apply jobs have been suspended, and
objects will not be resynced until recovery from Virtual Role Swap Test mode.
NOTE
The amount of time the recovery takes is in relation to the number of changes
made during the Virtual Test. The more changes made, the longer the recovery
takes. For example, if you plan on testing your day-end or month-end processes,
it could take a significant amount of time to recover. Vision recommends that
your Virtual Role Swap test be limited to one day.
17. Review Triggers on the backup node. Check to ensure the Triggers are set to the ❏
Preferred Primary status (4.23).
18. Review Constraints on all nodes. On the backup node, verify that the Constraints are ❏ ❏
set to the Preferred Primary status (4.24).

19. Optional - Do this step only if testing the communications. ❏
IMPORTANT
Perform this step only if performing the Virtual Role Swap with
Communications Test (which tests primary communications). If you choose to
do this, you will be required to restrict the production system so that users
cannot access it during the test. If you DO NOT want to test your primary
communications, skip this step and continue with the next step below.
NOTE
If you can create duplicate communications you can do your complete testing in
a Virtual Role Swap Test.
Perform the following steps manually:
a. End all the user jobs on the production server.

b. End the iTERA HA subsystem on all nodes.
c. Redirect the communications to the backup server.
d. Test to ensure communications are working properly.
20. Enable user profiles. You can do this for individual profiles by using the CHGUSRPRF ❏
command or enable all user profiles (5.1, F17=Enable Disabled Profiles).
21. Start User Subsystem and Jobs. Unless you have created your user up and user down ❏
programs, start user subsystem and user jobs on the backup node.
NOTE
We strongly encourage you to update the iTERA HA User Up and User Down
programs to automate the starting and ending of user jobs, subsystems, etc.
22. The target is now ready for application testing. Notify selected users that they can start ❏
testing applications.
NOTE
Test applications and verify that the license keys work.

End Virtual Role Swap; Resume Replication

End Virtual Role Swap Test - Resume Replication Pri Bkp
1. Optional - Do this step only if testing communications. ❏ ❏
IMPORTANT
Perform this step only if you executed the communications testing step in
the above procedure (as part of the Virtual Role Swap Test with
Communications Test). If you DID NOT test your primary
communications, skip this step and continue to step 2 below.
Perform the following steps manually:
a. End all the user jobs on the backup server.

b. Redirect communications to the primary server.
c. Start the iTERA HA subsystem on all nodes.
d. Test to ensure communications are working properly.
2. On the backup box, disable the user profiles that were enabled at the beginning ❏
of the test. You can do this for individual profiles using the CHGUSRPRF
command or use the mass change feature in User Profile Replication (5.1,
F15=Disable Profiles)
3. End all user jobs and subsystems. ❏ ❏

4. End the Virtual Role Swap Test and resume replication. On the backup, select ❏
40.30, F16=Role Maintenance, F19=Virtual Test. Review the Confirmation
screen. Enter Y if you want to run the iTERA HA Role Swap Primary End
program in the “Step 1” option. Otherwise, enter N and press F10.
The Role Swap Monitor will be displayed and processing will begin. Wait until
the Process Steps in the center section of the screen indicate a Complete message.
• The menus will no longer indicate that the system is in Virtual Role Swap Test
mode.
• Replication has resumed on the backup.
• The Heal process has identified any changed records and/or objects that have been
changed during the Virtual Test and has initiated heal requests.
NOTE
Changes made to objects that have been filtered using an IGNCHG filter will be
retained.
• The apply process will have begun to catch up.

• The IFS audit will identify changed objects and records and will correct the backup
when the audit is run.
NOTE
Be patient for the completion message to be displayed. The amount of time the
recovery takes is in relation to the number of changes made during the test. The
more changes made, the longer the recovery takes. If concerned about the amount
of time it is taking to recover, contact CustomerCare. To verify that the system has
caught up, execute the steps in “How to Check the Progress of the Virtual Role
Swap Recovery”, in the next section.
IMPORTANT
Spool files created during the Virtual Role Swap Test will not be removed from the
backup node.

5. If replicating WebSphere MQ, on the primary select option 5.6, F15=Start ❏

Replication to restart WebSphere MQ Replication. The MQ managers should
already be active.
6. Check Triggers on backup node. On the backup node verify that all Triggers have ❏
been set back to the correct status for the backup node (4.23).
7. Check Constraints on backup node. On the backup node verify that all ❏
Constraints have been set back to the correct status for the backup node (4.24).
8. Turn on Audit Command Console on both nodes using menu 6, F11=Start Auto ❏ ❏
Audit. As the screen directs, specify the desired time to begin the audits, then
press Enter. Otherwise, press F7 to start the audits immediately.
9. Release the Role Swap Readiness Monitor job on all nodes (E2SBS, opt 6 on ❏ ❏
xx_RSRMON).
10. On all nodes, release any jobs that were placed on hold prior to initiating the ❏ ❏
Virtual Role Swap Test.
11. From the Audit Command Console (6), select option 8=Execute Now on the ❏
IFS_AUDIT and SPLF_AUDP audits to submit them immediately. The status
of the audits should change to Run, then eventually to Done. If the status changes
to JobQ, go to the job queue and release the job.
12. Refresh the System Monitor on the primary node (1.1, F10=Update). Verify that ❏
there are no issues to resolve.

How to Promote Backup 2 to Backup 1

A Role Swap, Virtual Role Swap and Failover can only be performed on a
machine defined as BACKUP 1 in the CRG Recovery Domains screen (40.30,
F16). In order to perform these functions on a machine defined as BACKUP
2, it must be first promoted to BACKUP 1.
NOTE
A Replicate node cannot be used in a Role Swap or Virtual Role
Swap test.
Use the following procedure to promote a BACKUP 2 node to BACKUP 1.
Step Pri Bkp
1. Sign on to all nodes using the iTERA HA Admin profile. ❏ ❏

2. End the iTERA subsystem on all nodes (E2ENDSBS, primary first, then targets). ❏ ❏
3. On the primary, select menu option 30.21, F7=Resource Groups, opt 5=Recovery ❏
Set to display the CRG Recovery Domains screen.
4. On the primary, select option 8=Promote Current Role on the node defined as ❏
BACKUP 2. The Current Role field will display BACKUP 1. The node previously
defined as BACKUP 1 will now be defined as BACKUP 2.
NOTE
Disregard the message “Cluster resource group CRGnnnnn does not exist in
cluster CST” if it is displayed.
5. Optional: If the system that was promoted is to stay in the BACKUP 1 role, then ❏
change its Preferred Role to BACKUP 1 using option 9=Promote Preferred Role.
6. On all nodes, clear the following files by executing the commands: ❏ ❏
CLRPFM E2PXMONS
CLRPFM E2PXAMON
7. Sign off all nodes. ❏ ❏

8. Sign on all nodes using the iTERA HA Admin profile. ❏ ❏
9. Start the iTERA subsystem on all nodes (E2STRSBS; targets first, then primary). ❏ ❏
10. On the primary, check the System Monitor to ensure that all nodes display correctly ❏
(1.1, F10=Update Monitor).

How to Check the Progress of the Virtual Role

Swap Recovery
1. During the roll back portion of a Virtual Role Swap, the file E2PQAJ is
updated with the sequence number that the ZZ-Audit jobs are processing
about every 50-100 entries.
2. Compare the sequence number in the E2PQAJ file to the current sequence
number in the journal to estimate how much further the heal has to go.
a. On the backup machine select menu option 3.2.

b. Press F7 and verify that you are viewing the Local Receivers. This will
be displayed at the top middle portion of the screen. If it indicates
Remote, then Press the F7 until the display is correct.
c. Select option 5 on each one of the journals, then press Enter.
d. Press F15 to view the journal receivers. Scroll down until the last one in
the list is visible.
e. Select option 8 on the journals.
f. The current sequence number will be displayed at the bottom of the
screen. For each journal, write down the number and the journal
receiver name.
g. Run a query over the E2PQAJ file and compare the journal receiver
and the sequence number.
runqry *n e2pqaj
h. Verify that the number in the E2PQAJ is caught up with the local
receiver sequence that you wrote down.

How to Get and Apply iTERA
PTFs 10
Overview
All updates and enhancements for iTERA HA v6.0 and related products are
done with IBM-style PTFs. To successfully update iTERA HA, the PTFs
must be acquired from the Vision Solutions Support Central or FTP site,
then loaded and applied. This document provides step-by-step instructions
for these processes, as well as provides relevant supplementary material in the
appendices, including instructions for removing a PTF.
Product IDs
iTERA HA has two Product IDs:
• 7PA2K02 Cross Product Library (XP - Procedures library used by all

iTERA products).
• 7PA2K05 Base Product Library (iTERA HA).
IMPORTANT
You must retrieve, load, and apply PTFs for both product IDs to
be up to date with the core iTERA HA product.
IMPORTANT
Always retrieve and apply PTFs for the Cross Product library prior
to those for iTERA HA.
IMPORTANT
The subsystems must be ended on all CRGs while applying PTFs.
Additionally, if you have configured iTERA Alert, you must also periodically
load and apply PTFs for that product as well (Product ID 7PA2K25).
The following table shows products, product IDs, and versions that are
updated using these instructions.

Chapter 10: How to Get and Apply iTERA PTFs
Product or Product
Description Product ID Relevant Menu Options
Feature Version
Base iTERA HA 7PA2K05 V6R0M0 10.40 - Display

iTERA HA
product 10.45 - Retrieve and/or Install
Required for 7PA2K02 V4R3M0 10.41 - Display

iTERA HA and 10.46 - Retrieve and/or Install
iTERA Cross other Vision
Product Solutions products,
such as
RecoverNow
A configurable 7PA2K25 V6R0M0 10.42 - Display

iTERA Alert option within 10.47 - Retrieve and/or Install
iTERA HA.
NOTE
The document iTERA HA 6.0 PTF Service Pack Availability-nn.pdf
Service Pack Announcement document, available from Support
Central, includes the text of the PTF Cover Letters, as well as any
other special installation instructions. Additionally, the documents
iTERA HA v6.0 PTF Release Report.pdf and PTF Release Report
for Cross Product V4R3.pdf contain the summary details of all cover
letters to date and are also available for download from Support
Central.
How to Acquire iTERA PTFs

Although there are a few different ways to acquire iTERA HA PTFs, Vision
recommends using the menu options from within iTERA HA, which retrieves
the PTFs from the Vision Solutions FTP site. For detailed instructions, refer to
the section “Load and Apply PTFs” on page 237.
If, however, you are unable to access Vision Solutions FTP site from your IBM
i system, use one of the following alternate methods:
• Download PTFs from the Vision Solutions Support Central to your PC

then burn them to a CD or DVD (depending on file size). See “Download
PTFs to PC; Burn to Disc” on page 245.
• Obtain the ISO image and create a virtual optical device on the IBM i. See
“How to Create a Virtual Optical Drive on the IBM i and Download the
PTF ISO Image” on page 246.
• If you need only a few PTFs, use FTP from your PC to retrieve them from
the Vision Solutions FTP site, then transfer them to your IBM i. See

Load and Apply PTFs
“How to Use FTP to Retrieve an iTERA HA PTF to your PC Then

Transfer It to Your IBM i” on page 248.
• Regardless of the method you use to retrieve the PTFs, the instructions in
the section “Load and Apply PTFs” on page 237 should be used to load
and apply them.
Load and Apply PTFs

iTERA HA menu options provide the capability to retrieve, load, and apply
PTFs from Vision Solutions’ FTP site directly from your IBM i (if your system
has access to the Internet), from another IBM i, or from CD/DVD.
The menu options to retrieve and install PTFs (listed in the table above) call
the ITINSTPTF command, and pre-populate some of the required
parameters, such as product ID and product release.
IMPORTANT
The instructions describe the process for the Cross Product first.
Load and apply PTFs for the Cross Product completely prior to
loading and applying PTFs for iTERA HA. If using iTERA Alert,
load and apply PTFs for that product after iTERA HA PTFs have
been applied.
You may load and apply Cross Product PTFs on all nodes
simultaneously, then subsequently repeating the instructions for
iTERA HA PTFs, and then again for iTERA Alert PTFs.
Do not start the subsystems until PTFs for both Cross Product and
iTERA HA have been applied on all nodes in the environment.

1. Sign on to your system (any node) as the iTERA HA user. Select menu
option 10.46 to display the ITINSTPTF command with the parameters
entered for the Cross Product. The following screen is displayed:
IMPORTANT
When repeating these instructions, use menu option 10.45 for
iTERA HA PTFs and 10.47 for iTERA Alert PTFs. These menu
options display the ITINSTPTF screen with Product ID and Release
parameters pre-populated.
2. Set the parameters according to the following table, and press Enter. See
screen below for example.
Parameter Description
Set to one of the following:
iTera Product ID
• 7PA2K02 for cross-product PTFs
• 7PA2K05 for iTERA HA
• 7PA2K25 for iTERA Alert
iTera Product Release

• V4R3M0 for cross-product PTFs
• V6R0M0 for iTERA HA
• V6R0M0 for iTERA Alert

Load and Apply PTFs
Parameter Description
Enter the location of the PTFs:

• *ITERA - The PTFs are located on the Vision
Solutions FTP server and are automatically
retrieved. To use this option, the FTP options
must be configured. See F10=Additional
parameters, below.
Location of PTF Files • *RMTSYS - The PTFs are on a remote system.
(After setting the other parameters and pressing
Enter, the Remote System parameters will be
displayed.)
• *OPT - The PTFs are on a CD or Virtual Optical
Device. For more information on obtaining a CD
of PTFs, see “Download PTFs to PC; Burn to
Disc” on page 245.
Get PTF List Enter *YES to get the list of PTFs.
Display PTFs not on Enter *YES to display the PTFs that are not currently
System on the system.
Get PTF Save Files Enter *YES to get the PTF save files.
Load and Apply PTFs Set to *NO.

• Set to *NO to keep the PTF save files once the
PTFs have been permanently applied. Use this
setting when generally applying PTFs.
Remove save files for • Set to *YES to automatically delete the PTF save
perm apy files once the PTFs have been permanently
applied. This check is executed each time the
ITINSTPTF command is used (including menu
options 10.45, 10.46, 10.47). See “Important
Information: Permanently Applying PTFs” on
page 244.
Indicate whether to display PTFs that are not

Execute DSPPTF? currently on the system.
Set to *YES.
If *RMTSYS was entered in Location of PTF files,

Remote System
enter the IP address of the remote IBM i.
Remote System User If *RMTSYS was entered in Location of PTF files,

ID enter the iTERA HA user ID for the remote IBM i.
If *RMTSYS was entered in Location of PTF files,

Remote User
Password enter the iTERA HA user password for the remote
IBM i.

NOTE
Additional details on each parameter are available in the help text
(F1).
F10=Additional Parameters
When *ITERA is specified in the Location of PTF Files parameter, the PTFs
will be retrieved from Vision’s FTP server. Normally, these parameters are
defined when iTERA HA is initially configured and the settings are retained. If
not already configured, specify the following:
• iTera FTP user id / iTera FTP password - Enter the user ID “public”
and password provided by Vision Solutions CustomerCare.
• iTera FTP server address - Enter the Vision Solutions FTP server IP
address:166.70.109.194.
• Include “sendpasv” - “sendpasv” is an FTP subcommand. If you
have firewall issues, enter *YES for this parameter.

Load and Apply PTFs
3. Upon pressing Enter, if your system can successfully retrieve PTFs, the
following report is displayed (the actual list of PTFs will vary):
This report displays a list of PTFs that are available for download that are
not already installed on your IBM i. If the list is empty, PTFs are up to date
for this product ID (i.e., no PTFs are displayed because they have all been
applied).
4. Press Enter to continue.
If there are any unapplied PTFs, status messages are displayed indicating
that they are being transferred to your IBM i (e.g., Getting PTF
0XP0009...).
5. Enter menu option 10.41 (Cross Product) to display the PTFs retrieved in
the previous step.
NOTE
Use menu option 10.40 for iTERA HA PTFs and 10.42 for iTERA
Alert PTFs.
NOTE
The document iTERA HA 6.0 PTF Service Pack Availability-nn.pdf
Service Pack Announcement document, available from Support
Central, includes the text of the PTF Cover Letters, as well as any
other special installation instructions. Additionally, the documents
iTERA HA v6.0 PTF Release Report and PTF Release Report for
Cross Product V4R3 contain the summary details of all cover letters
to date and are also available for download from Support Central.

6. Sign off and end the iTERA subsystems:
a. Have all iTERA HA users sign off all nodes. (No users can access
any menus while PTFs are being applied.)
b. End the iTERA HA subsystem on all nodes (primary first, then
targets).
c. Sign off all nodes.
d. Sign on to all nodes with the iTERA HA Admin profile.
IMPORTANT
After you sign back on, do not select any iTERA HA menu options
or use any iTERA HA commands other than 10.45, 10.46, and
10.47. Doing so may allocate or lock iTERA HA objects which
could cause problems during the iTERA HA PTF apply process.
7. Enter menu option 10.46. Change the value in the Load and Apply PTFs
field to *YES, as indicated, then press Enter.
NOTE
The example above displays the Cross Product ID and Release
information. To apply iTERA HA PTFs, use menu option 10.45. To
apply iTERA Alert PTFs, use menu option 10.47.
Status messages displayed at the bottom of the screen indicate the PTFs
are being loaded and applied. The PTF files are placed in library
QGPL.

Load and Apply PTFs
8. Enter menu option 10.41 and press Enter. Verify that the Cross Product
PTFs were correctly applied.
NOTE
Use 10.40 for iTERA HA PTFs and 10.42 for iTERA Alert PTFs.
NOTE
The appropriate status for most PTFs is Temporarily Applied.
PTFs with the status Not Applied indicate that the load process
failed. These will need to be investigated and resolved.
Other statuses that may be displayed are Superseded, or Permanently

Applied.
9. Repeat step 1 through step 8 for iTERA HA PTFs (menu option 10.45;
iTERA Product ID 7PA2K05, Release V6R0M0).
10. Repeat step 1 through step 8 for iTERA Alert PTFs (menu option 10.47;
iTERA Product ID 7PA2K25, Release V6R0M0).
11. If you did not apply PTFs on the other nodes simultaneously, repeat step 1
through step 10 on each of the other nodes in the iTERA HA
environment.
12. After the PTFs have been successfully applied, sign off, then sign on using
the iTERA HA Admin profile.
13. Start the iTERA HA subsystems on all nodes (E2STRSBS; targets first,
then primary).

Important Information: Permanently Applying PTFs

Before you upgrade the operating system on your IBM i system, you must
apply all iTERA PTFs permanently. Failure to do so could result in losing
some or all of the iTERA PTFs.
1. Have all iTERA HA users sign off all nodes, then end the iTERA HA
subsystem on all nodes (primary first, then target).
2. Sign off all nodes, then sign back on to all nodes with the iTERA HA user
profile.
3. Execute the following commands:
APYPTF LICPGM(7PA2K02) RLS(V4R3M0) APY(*PERM) DELAYED
(*NO)

(*NO)
If using iTERA Alert, also execute the following command:

(*NO)
4. After the upgrade, menu options 10.46 and 10.45 should be executed to
remove the save files for the permanently applied PTFs using the following
parameters.
For Cross Product:
For iTERA HA:
5. Optional: If using iTERA Alert, execute 10.47, setting the parameters as

above. This menu option specifies 7PA2K25 in the iTERA Product ID
field.

Other PTF Procedures
Keep in mind that PTFs that have been permanently applied cannot be
removed, so it is a good idea to apply them permanently only when an OS
upgrade is executed.
IMPORTANT
Do not permanently apply iTERA PTFs via an IPL. If you want to
permanently apply PTFs for iTERA products, they must be applied
as indicated above. Permanently applying iTERA PTFs during an
IPL will result in the IPL hanging because the iTERA libraries will
not be in the library list.
Download PTFs to PC; Burn to Disc

If you cannot access Vision Solutions’ FTP server from your IBM i, but you
can from your PC, use these instructions. These instructions guide you
through downloading and burning the latest iTERA HA PTF ISO image.
IMPORTANT
Do not just copy the PTF *.iso file onto a disc. You must use the
“Burn Image to Disk” option of the burning software. If you don’t
have CD burning software, there is a free ISO image plug-in for
Windows at http://isorecorder.alexfeinman.com/
Download the iTERA HA PTF CD Image From Vision

Solutions Support Central Site.
1. Go to http://portal.visionsolutions.com/extlogin.aspx.
2. Enter your user ID and password then select Login. If you do not have a
username and password, select Sign up.
3. Click “iTERA HA” from the My Products section in the left panel.
4. Click the “Customer Care” link in the center of the page.
5. Verify that 6.0 is displayed in the Version drop-down box, then select
Downloads.
6. Download itha60_ptf.zip to your PC.
IMPORTANT
The PTF file may be large. Download time will depend on your
connection speed.

7. After file has finished downloading, unzip it, then burn the *.iso image file
to disc (CD or DVD, depending on file size).
8. Once you have burned the installation media to disc, follow the
instructions in the chapter “Load and Apply PTFs” on page 237. For step
2, load the disc on the IBM i (ensure there are no other discs loaded) and
specify the disc’s location in step 2 of that section.
How to Create a Virtual Optical Drive on the IBM i

and Download the PTF ISO Image
This procedure is an alternative to creating a disc using disc burning software.
This procedure may be used if you can FTP from the IBM i to the Internet.
You will create a virtual optical device on the IBM i then FTP the iTERA HA
PTF CD image (ISO image) from the Vision Solutions FTP site to the IBM i.
1. Make certain you are signed on with the iTERA HA User Profile so that
you have all required authorities.
2. Create a virtual optical device by entering the command:
CRTDEVOPT DEVD(OPTVRT01) RSRCNAME(*VRT)
3. Vary on the virtual optical device. To make the device active, enter:
VRYCFG CFGOBJ(OPTVRT01) CFGTYPE(*DEV) STATUS(*ON)
4. Create an image catalog by entering the command:

CRTIMGCLG IMGCLG(OPTVRT01)
DIR('/home/optvrt01/Distribution/7PA2K05/V6R0M0/6.0.
nn.00/ISO Files')
(where nn is the current PTF Service Pack number)
5. Download the zip file of the ISO image to the IBM i. Type the following
commands, pressing enter after each line.
FTP ‘166.70.109.194’
Login with user name public and use the password provided by
Vision Solutions CustomerCare.
nam 1
NOTE
Ignore any error that may occur on this step.
bin
lcd /home/optvrt01

cd 7PA2K05/V6R0M0 (case-sensitive)
DIR
(view the list of files)

get itha60_ptf.zip
(case-sensitive)
When download is complete:

quit
6. Unzip the downloaded ISO image. On the command line, type the
following commands:
QSH
CD /HOME/OPTVRT01
JAR XF ITHA60_PTF.ZIP
The file will be unzipped to the following location:
/home/optvrt01/Distribution/7PA2K05/V6R0M0/6.0.14.00/ISO
Files/itha60_ptf.iso
7. Add an image catalog entry for the iso image. Type the command:
ADDIMGCLGE IMGCLG(OPTVRT01)
FROMFILE('/home/optvrt01/Distribution/7PA2K05/V6R0M0
/6.0.nn.00/ISO Files/itha60_ptf.iso')
8. Load the image catalog. Use the command LODIMGCLG. This associates
the virtual optical device to the image catalog.
LODIMGCLG IMGCLG(OPTVRT01) DEV(OPTVRT01)
9. Verify it is mounted by typing WRKOPTVOL. You should see the ISO

mounted in OPTVRT01.
The image catalog is ready for use. You can now load and apply PTFs using the
instructions in the section “Load and Apply PTFs” on page 237. Make certain
to specify *OPT in the Location of PTF files field (i.e., this will point to the
optical device you just created) to retrieve PTFs.

How to Use FTP to Retrieve an iTERA HA PTF to

your PC Then Transfer It to Your IBM i
NOTE
The instructions in this section should only be used if you need just a
few PTFs and your IBM i cannot FTP files from the FTP server but
your PC can. If you need to apply a large number of PTFs, use the
instructions in the section “Download PTFs to PC; Burn to Disc” on
page 245.
1. Create a directory on your PC to hold iTERA HA PTFs. In the following

examples we will use a PC directory named iTERAPTFs.
2. Click on the Windows “Start” button and select “Run”. Type cmd in the
Open field. Click OK. A dialogue box with text similar to the following
appears:
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
3. If you wish to download the PTF(s) to a different directory than shown on

the prompt line, use the cd command to change to a different local
directory.
4. At the “C:\” prompt in the dialogue box, type the following command
then press Enter:
ftp ftp.iterainc.com
5. If the connection is successful, you will be prompted for a user name. Type
public and press enter.
6. Type the password obtained from CustomerCare and press Enter.

7. On the ftp prompt, type the following commands and press Enter after
each command:
bin
cd 7PA2K05/V6R0M0
dir
(this will list the available PTF save files; the cover letter save files end
with “CL”)
quote site na 1
(Ignore the “SITE NA not understood” error message if you get it.)

8. Type the following command for each of the iTERA HA PTFs you wish
to retrieve and press enter:
get Q1HAnnnn c:\iTERAPTFs\Q1HAnnnn.savf
(where nnnn is the four digit PTF number; If you did not create the
“iTeraPTFs” directory, replace iTERAPTFs with the directory name
you created.)
9. To retrieve iTERA Cross Product library PTFs, type the following and
press enter.
cd /7PA2K02/V4R3M0
dir
10. Type the following command for each of the Cross Product Library PTFs
you wish to retrieve and press enter:
get Q0XPnnnn c:\iTERAPTFs\Q0XPnnnn.savf
(where nnnn is the four digit PTF number).
11. To end the FTP session, type quit and press enter.
12. At the "C:\" prompt in the dialogue box, type this command:
ftp nnn.nnn.nnn.nnn
(where nnn.nnn.nnn.nnn is the IP address (or system name) of your

IBM i) and press Enter.
13. Type your IBM i user profile and password as prompted.
14. Type the following commands; press Enter after each:

bin
cd qgpl
Na 1
15. Type one of the following:
put c:\iTERAPTFs\Q1HAnnnn.savf
or
put c:\iTERAPTFs\Q0XPnnnn.savf
(Depending on whether your are downloading HA or XP PTFs, and
where nnnn is the PTF number.)

NOTE
When you FTP a file with an extension of “.savf ” to the IBM i, it
automatically assumes it is a save file, and you do not need to first
create a save file to receive the data.
16. Repeat step 15 for each additional iTERA HA PTF you wish to send to
your IBM i.
17. To end the FTP session, type quit and press enter.
18. After you have the iTERA HA PTF save file on your IBM i, you are ready
to load it. (Loading the PTF does not make any changes to your system.
It simply registers the PTF to the operating system and stages it for the
actual applying of the changes.)
PTFs are loaded into i5/OS using the LODPTF command. Type LODPTF
on the command line and press F4 to prompt.
Fill in the prompts indicated in the following screen then press Enter:
Load Program Temporary Fix (LODPTF)

Product . . . . . . . . . . . . > 7PA2K02 F4 for list

Device . . . . . . . . . . . . . > *SAVF Name, *SERVICE, *SAVF
PTF numbers to select . . . . . > 0xxnnnn Character value, *AL
+ for more values _______
PTF numbers to omit . . . . . . _______ Character value
+ for more values _______
Superseded PTFs . . . . . . . . *APYPERM *APYPERM, *NOAPY
Release . . . . . . . . . . . . V4R3M0 *ONLY, VxRyMz
Save file . . . . . . . . . . . q0xxnnnn Name
Library . . . . . . . . . . . qgpl Name, *LIBL, *CURLIB
Bottom
F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this display
NOTE
To load iTERA HA PTFS, use 7PA2K05 for the Product ID and
V6R0M0 for the Release. The Save File name starts with Q1.
NOTE
“xx” is either “HA” for iTERA HA PTFs or “XP” for Cross Product
Library PTFs; “nnnn” is the number of the PTF you wish to load.
Repeat this step for each PTF you wish to load.
19. To verify that the PTFs were loaded, use menu option 10.41 (Cross
Product PTFs), 10.40 (iTERA HA PTFs), or 10.42 (iTERA Alert PTFs)
on a command line and press enter. A screen similar to the following is
displayed:

Display PTF Status

System: IT800
Product ID . . . . . . . . . . . . . : 7PA2K02
IPL source . . . . . . . . . . . . . : ##MACH#B
Release of base option . . . . . . . : V4R3M0
Type options, press Enter.

5=Display PTF details 6=Print cover letter 8=Display cover letter
PTF IPL
Opt ID Status Action
_ 0XP0009 Not applied None
_ 0XP0008 Temporarily applied None
_ 0XP0007 Temporarily applied None
Bottom
F3=Exit F11=Display alternate view F17=Position to F12=Cancel
NOTE
If the PTF was loaded properly, it will appear in the list with a status
of “Not applied”.
20. After you have the iTERA HA PTF save file(s) on your IBM i and the
PTFs loaded, you are now ready to apply them. Applying the PTF will
make changes to iTERA HA.
a. Have all iTERA HA users sign off all nodes.

b. End the iTERA HA subsystem on all nodes (primary first, then target).
c. Sign off all nodes.
d. Sign on to all nodes with the iTERA HA user profile.
21. Use the APYPTF command to apply the PTF(s).
To apply all loaded PTFs, type the following on the command line and
press enter:
For iTERA Cross Product PTFs use:

APYPTF LICPGM(7PA2K02) RLS(V4R3M0)
For iTERA HA PTFs use:

APYPTF LICPGM(7PA2K05) RLS(V6R0M0)
To apply a single specific PTF, type the following on the command line
and press enter:
• For iTERA Cross Product PTFs use:
APYPTF LICPGM(7PA2K02) RLS(V4R3M0) SELECT(0XPnnnn)
• For iTERA HA PTFs use:

APYPTF LICPGM(7PA2K05) RLS(V6R0M0) SELECT(1HAnnnn)
(nnnn is the four digit PTF number)

22. To verify that the PTFs were applied, enter menu option 10.41 (Cross
Product PTFs) or 10.40 (iTERA HA PTFs). A screen similar to the
following is displayed:
Display PTF Status

System: IT800
Product ID . . . . . . . . . . . . . : 7PA2K05
IPL source . . . . . . . . . . . . . : ##MACH#B

PTF IPL
_ 1HA0009 Temporarily applied None
Bottom
F3=Exit F11=Display alternate view F17=Position to F12=Cancel
NOTE
If the PTF was applied properly, it will appear in the list with a
status of Temporarily applied, or Superseded. (In some cases the
status may be Permanently Applied.) PTFs with the status Not
Applied indicate that the load process failed and they will need to be
resolved.
23. After the PTFs are successfully applied, do the following:
a. Sign off then sign on to all nodes using the iTERA HA user profile (to
release locks and re-establish the proper library list).
b. Start the iTERA HA subsystem on all nodes (E2STRSBS; target first,
then primary).
Troubleshooting and Maintenance

The following procedures can be used to troubleshoot issues with iTERA
PTFs.
Errors
Unable to find PTF list file IPT7PA2K05…
If you receive the screen message “Unable to find PTF list file
IPT7PA2K05…” or similar, the simplest resolution is to ensure that the PTFs
for 7PA2K02 have been retrieved and applied prior to attempting to retrieve
PTFs for 7PA2K05. This will, in most cases, resolve the problem. If you still
receive the error, contact CustomerCare.

How to Remove or Correct an iTERA PTF

If an iTERA PTF needs to be removed (perhaps due to a Technical Alert
indicating a PTF should be removed) or if you encounter problems during the
PTF apply process, there are two IBM commands you may use to remove or
correct an iTERA PTF.
• RMVPTF removes the new objects applied to the iTERA HA product

from the PTF and restores original objects back to the appropriate
libraries.
• DLTPTF deletes the PTF save files (PTF and cover letter) from the QGPL
library.
Remove PTF:
Use the RMVPTF command to return the PTF environment (programs, etc.) to
the previous level (the level prior to applying the PTF).
NOTE
Note: The Extent of Change parameter must be set to *PERM in
order to prevent problems when reapplying the PTF.
For Cross Product PTFs:
Remove Program Temporary Fix (RMVPTF)
Product . . . . . . . . . . . . 7PA2K02 F4 for list

Release . . . . . . . . . . . . V4R3M0 *ONLY, VxRxMx
PTF numbers to select . . . . . 0XPnnnn Character value, *ALL
+ for more values
PTF numbers to omit . . . . . . Character value
+ for more values
Extent of change . . . . . . . . *PERM *TEMP, *PERM
Delayed PTFs . . . . . . . . . . *NO *NO, *YES
NOTE
PTF number example: 0XPnnnn (where nnnn is the PTF number).

For iTERA HA PTFs:
Remove Program Temporary Fix (RMVPTF)

PTF numbers to select . . . . . 1HAnnnn Character value, *ALL
+ for more values
+ for more values
Extent of change . . . . . . . . *PERM *TEMP, *PERM
Delayed PTFs . . . . . . . . . . *NO *NO, *YES
NOTE
PTF number example: 1HAnnnn (where nnnn is the PTF number).
Use the DLTPTF command delete the save files for the PTF:
Delete Program Temporary Fix (DLTPTF)

PTF . . . . . . . . . . . . . . 0XPnnnn Character value, *ALL
+ for more values
Release . . . . . . . . . . . . V4R3M0 *ALL, VxRxMx
Delete duplicate PTF numbers . . *NO *YES, *NO
NOTE
PTF number example: 0XPnnnn or 1HAnnnn (where nnnn is the
PTF number).
NOTE
For iTERA HA PTFs: Use 7PA2K05 and V6R0M0 for the Product
and Release (the example at the left is for the Cross Product.
Check to see if the PTF was removed. Enter menu option 10.41 (Cross
Product PTFs) or 10.40 (iTERA HA PTFs).
Delete a PTF that has problems applying during the PTF apply
process:
Use this procedure if, during the PTF apply process (NOT the PTF retrieve
process), the status for the PTF indicates *Savefile.
Use the DLTPTF command to delete the PTF. Using this command will delete
the save files for the PTF from QGPL.

NOTE
Prior to the DLTPTF command, you may execute the RMVPTF
command for this PTF but this is not required since the PTF was
never applied nor the iTERA HA product upgraded.
Delete Program Temporary Fix (DLTPTF)
PTF . . . . . . . . . . . . . . 0XP0100 Character value, *ALL

+ for more values
Release . . . . . . . . . . . . V4R3M0 *ALL, VxRxMx
Delete duplicate PTF numbers . . *NO *YES, *NO
Specify the following parameters:
PTF
Product Area Number Product ID Release
Cross Product (XP) 0XPnnnn 7PA2K02 V4R3M0
iTERA HA 1HAnnnn 7PA2K05 V6R0M0
iTERA Alert 0EAnnnn 7PA2K25 V6R0M0
• Use the default values for the remaining parameters.
Retrieve, load, and apply the PTF again. Should the PTF not apply again (i.e.,
status indicates *Save file only), follow the steps below.
Use the IBM command LODPTF to load the PTF (register the PTF to the
operating system).
Load Program Temporary Fix (LODPTF)

Device . . . . . . . . . . . . . *SERVICE Name, *SERVICE, *SAVF
PTF numbers to select . . . . . > 0xp0052 Character value, *ALL
+ for more values
+ for more values
Superseded PTFs . . . . . . . . *APYPERM *APYPERM,*NOAPY
Release . . . . . . . . . . . . V4R2M0 *ONLY, VxRyMz
Sequence number . . . . . . . . *SEARCH 1-16777215, *SEARCH
End of media option . . . . . . *REWIND *REWIND, *LEAVE, *UNLOAD
Path identifier . . . . . . . . *FIRST 1-9999, *FIRST, *SELECT
Save file . . . . . . . . . . . Name
Library . . . . . . . . . . . *LIBL Name, *LIBL, *CURLIB
Copy PTF cover letter . . . . . *YES *YES, *NO, *ONLY

Specify the following parameters:
PTF
Product Area Product ID Release
Number
Enter menu option 10.41 (Cross Product PTFs) or 10.40 (iTERA HA PTFs)
to check the PTF status.
Display PTF Status

System: IT800L
Product ID . . . . . . . . . . . . . : 7PA2K02
IPL source . . . . . . . . . . . . . : ##MACH#B
PTF IPL
0XP0052 Not applied None
0XP0051 Temporarily applied None
0XP0045 Permanently applied None
More...
NOTE
If the status indicates Not applied, the PTF loaded correctly. If this
is not the case, contact CustomerCare.
Have all iTERA HA users sign off all nodes. End the iTERA HA subsystem on
all nodes (primary first, then target). Sign off all nodes. Sign on to all nodes
with the iTERA HA user profile.

Use the IBM command APYPTF to apply (upgrade the iTERA HA product)
the PTF, specifying the parameters indicated.
Apply Program Temporary Fix (APYPTF)

PTF numbers to select . . . . . 0XP0052 Character value, *ALL
+ for more values
+ for more values
Extent of change . . . . . . . . *TEMP *TEMP, *PERM
Delayed PTFs . . . . . . . . . . *NO *NO, *YES, *IMMDLY
IPL apply options:
Apply at unattended IPL . . . *YES *NO, *YES
Prerequisite lic int code . . *APYPERM *APYPERM, *NOAPY
Apply requisite PTFs . . . . . . *NO *NO, *YES
PTF
Product Area Product ID Release
Number
Display the PTF status; 10.41 (Cross Product), or 10.40 (iTERA HA).
Display PTF Status

System: IT800L
Product ID . . . . . . . . . . . . . : 7PA2K02
IPL source . . . . . . . . . . . . . : ##MACH#B

PTF IPL
0XP0051 Superseded None
0XP0045 Permanently applied None
More...
IMPORTANT
If the status indicates Superseded or Temporarily applied, the PTF
loaded correctly. If this is not the case, contact CustomerCare.

After the PTFs are successfully applied, sign off then sign on to all nodes using
the iTERA HA user profile (to release locks and re-establish the proper library
list), then start the iTERA HA subsystem on all nodes (E2STRSBS; target first,
then primary).

How to Automate PTF Retrieval and Notification

If at least one of your IBM i machines has iTERA HA installed and can access
Vision Solutions’ FTP site, you can fully automate the retrieval and
notification of iTERA HA PTFs (for both the iTERA Cross Product library
and iTERA HA).
1. Compile a program named E2GETPTFS into the ITHAxx library from

the following CLLE source (where “xx” is a valid Resource Group Code).
/*========================================================================*/
/* This program will retrieve PTFs from Vision Solutions’ FTP site and */
/* put a notification message into the iTera HA message log for each */
/* unapplied PTF found on your system. It should be run on a periodic */
/* basis from a scheduler (i.e. monthly) */
/* */
/* Note: PTFs are NOT applied by this process, they are only retrieved. */
/* After you are notified that they need to be applied, you are */
/* responsible to review and apply them according to the instructions */
/* contained in the cover letter(s). */
/*========================================================================*/
PGM
DCL VAR(&JobName) TYPE(*CHAR) LEN(32)
DCL VAR(&MsgDesc) TYPE(*CHAR) LEN(25)
DCL VAR(&Pgm) TYPE(*CHAR) LEN(10) +
VALUE('E2GETPTFS')
DCLF FILE(QADSPPTF)
/* Fetch PTFs for the iTera Cross Product Library */
ITINSTPTF PRDID(7PA2K02) RELEASE(V4R3M0) GET(*YES) +
GETSAVF(*YES)
/* Fetch PTFs for iTera HA */
ITINSTPTF PRDID(7PA2K05) RELEASE(V6R0M0) GET(*YES) +
GETSAVF(*YES)
DLTF FILE(QTEMP/NONAPYPTFS)
MONMSG MSGID(CPF0000)
DSPPTF LICPGM(7PA2K02) SELECT(*ALL) RLS(V4R3M0) +
OUTPUT(*OUTFILE) +
OUTFILE(QTEMP/NONAPYPTFS) OUTMBR(*FIRST *ADD)
OUTPUT(*OUTFILE) +
OVRDBF FILE(QADSPPTF) TOFILE(QTEMP/NONAPYPTFS)
PRIMEREAD:
RCVF RCDFMT(QSCPTF)
MONMSG MSGID(CPF0864) EXEC(GOTO CMDLBL(END))
MAIN:
IF COND(&SCSTATUS *EQ 'Damaged' *OR &SCSTATUS +
*EQ 'Save file only' *OR &SCSTATUS *EQ +
'Not applied') THEN(DO)
CHGVAR VAR(&JOBNAME) VALUE('Z$GETPTFS found new +
PTF' *BCAT &SCPTFID)
CHGVAR VAR(&MsgDesc) VALUE('Please review and apply')
CALL PGM(E22090RP) PARM(&JOBNAME &MsgDesc &Pgm)
ENDDO
READNEXT:
RCVF RCDFMT(QSCPTF)
GOTO CMDLBL(MAIN)
END:
ENDPGM
If you do not have the ability to access our FTP site from your target
node, you can create this CLLE source on the target node with the
following changes. This will enable the target node to bring over all the
new PTFs that have been retrieved to the primary node.

Compile a program named E2GETPTFS into the ITHAxx library on

the target node from the following CLLE source (where “xx” is a valid
Resource Group Code).
/*========================================================================*/
/* This program will retrieve PTFs from iTera's FTP site and put a */
/* notification message into the iTera HA message log for each unapplied */
/* PTF found on your system. It should be run on a periodic basis */
/* from a scheduler (e.g. monthly). */
/* */
/* Note: PTFs are NOT applied by this process, they are only retrieved. */
/* After you are notified that they need to be applied, you are */
/* responsible to review and apply them according to the instructions */
/* contained in the cover letter(s). */
/*========================================================================*/
PGM
DCL VAR(&JobName) TYPE(*CHAR) LEN(32)
DCL VAR(&MsgDesc) TYPE(*CHAR) LEN(25)
DCL VAR(&Pgm) TYPE(*CHAR) LEN(10) +
VALUE('E2GETPTFS')
DCLF FILE(QADSPPTF)
/* Fetch PTFs for the iTera Cross Product Library */
ITINSTPTF PRDID(7PA2K02) RELEASE(V4R3M0) LOC(*RMTSYS)

RMTSYS('IP address of primary') RMTUSER(iTera HA User profile) RMTPWD(iTera HA
password for User profile)
/* Fetch PTFs for iTera HA */
ITINSTPTF PRDID(7PA2K05) RELEASE(V6R0M0) LOC(*RMTSYS)

RMTSYS('IP address of primary') RMTUSER(iTera HA User profile) RMTPWD(iTera HA
password for User profile)
DLTF FILE(QTEMP/NONAPYPTFS)
MONMSG MSGID(CPF0000)
OUTPUT(*OUTFILE) +
OUTPUT(*OUTFILE) +
OVRDBF FILE(QADSPPTF) TOFILE(QTEMP/NONAPYPTFS)
PRIMEREAD:
RCVF RCDFMT(QSCPTF)
MAIN:
IF COND(&SCSTATUS *EQ 'Damaged' *OR &SCSTATUS +
*EQ 'Save file only' *OR &SCSTATUS *EQ +
'Not applied') THEN(DO)
CHGVAR VAR(&JOBNAME) VALUE('Z$GETPTFS found new +
PTF' *BCAT &SCPTFID)
CHGVAR VAR(&MsgDesc) VALUE('Please review and apply')
CALL PGM(E22090RP) PARM(&JOBNAME &MsgDesc &Pgm)
ENDDO
This program can then be put into the job scheduler on the target
node. Schedule this job later than this same job on the primary. Allow
time for the primary node time to finish the download before the
process is started on the target.
2. Call this program on a periodic basis from any scheduler (we recommend
once a month). The following is an example of how to schedule the job in
the IBM Job Scheduler using the ADDJOBSCDE command:

NOTE
In this example, a job has been set up to run on the second Monday of each
month at 11 AM.
Add Job Schedule Entry (ADDJOBSCDE)
Job name . . . . . . . . . . . . > Z$GETPTFS Name

Entry number . . . . . . . . . . > 000027 000001-999999, *ONLY
Command to run . . . . . . . . . CALL PGM(E2GETPTFS)_________
____________________________________________________________________
Frequency . . . . . . . . . . . *MONTHLY *SAME, *ONCE, *WEEKLY...
Schedule date . . . . . . . . . *NONE Date, *SAME, *CURRENT...
Schedule day . . . . . . . . . . *MON *SAME, *NONE, *ALL, *MON
Schedule time . . . . . . . . . '11:00:00' Time, *SAME, *CURRENT
Relative day of month . . . . . 2 *SAME, *LAST, 1, 2, 3, 4
Job description . . . . . . . . E2JOBD Name, *SAME, *USRPRF
Library . . . . . . . . . . . ITHAxx Name, *LIBL, *CURLIB
Job queue . . . . . . . . . . . E2JOBQ Name, *SAME, *JOBD
Library . . . . . . . . . . . ITHAxx Name, *LIBL, *CURLIB
User . . . . . . . . . . . . . . E2xxXUSER Name, *SAME, *JOBD,
Message queue . . . . . . . . . *USRPRF Name, *SAME, *USRPRF,
Library . . . . . . . . . . . __________ Name, *LIBL, *CURLIB
Text 'description' . . . . . . . 'Get iTera HA PTFs Every 2nd Monday'
If the process finds any PTFs that you have not yet applied to your iTERA HA
application, it will retrieve them (via FTP) and write a message to the iTERA
HA message log notifying you that a new PTF was retrieved.
The iTERA HA message log messages will display similar to the following:
Display Report
Report width . . . . . : 99
Position to line . . . . . Shift to column . . . . . .
Line ....+....1....+....2....+....3....+....4....+....5....+....6....+....7
MLJOB MLDESC MLPGM
000001 Z$GETPTFS found new PTF 0XP0018 Please review and apply E2GETPTFS
000002 Z$GETPTFS found new PTF 0XP0016 Please review and apply E2GETPTFS
000003 Z$GETPTFS found new PTF 1HA0075 Please review and apply E2GETPTFS
****** ******** End of report ********


iTERA HA Scheduled Audits and
Other Jobs A
The following iTERA HA audits and other jobs may be scheduled to run in
the IBM Job Scheduler. Other job scheduling software may be used, but it is
not supported by Vision Solutions.
From the Audit Command Console (menu 6), select option 2 for the audit,
then select F7=Create Job. The Add Job Schedule Entry screen is displayed.
Make adjustments as needed, then press Enter to add the scheduled entry.
When the Audit Command Console is turned on, it will automatically

initiate the job if for some reason the audit has not run within the maximum
allowable time.
IMPORTANT
The name of the job in the job scheduler must be entered as
indicated in the table below in order for the Audit Command
Console to update the status correctly.
Replace “xx” with the two-character CRG code. For example, if the
CRG code is A1, then for job E2xxAAUDLV, enter
E2A1AAUDLV for the job name in the job scheduler.
• Audits are assigned a replication option (group). It is better if only one

audit from a replication group run at a time. You can run as many audits
concurrently from different replication groups as you like.
• If the audit has not run within the maximum allowable time, the Audit
Command Console will automatically initiate the job.
• The Audit Command Console will not display audits for the
components that are not enabled in the Replication Options 30.23
screen (e.g., Job Scheduler Replication, Spool File Replication, etc.)
unless F8=List All Audits is used.

Appendix A: iTERA HA Scheduled Audits and Other Jobs
Name of
Audit Required? Job Scheduler Entry Job in Job Scheduling Notes
Scheduler
Alternate Name Audit Daily on primary CALL PGM(E21315CPS) PARM(*FIX E2xxALTNP

ALT_NAMEP *ALL *ALL)
Audit Level Audit Weekly on all CALL PGM(E21522CP) E2xxAAUDLV

ALVL_AUDIT nodes
Audit Stream Daily on primary CALL PGM(E21399CPS) E2xxASTRM The audit stream runs the following
AUDSTREAM audits, in order:
CHKOBJMTCH (Check Object Match)
JRNOBJLST (Journaled Object List)
JRNOBJJRN (Journal to Object Audit)
RCDCNTCHG (Record Count –
Changed Objects)
If the Audit Stream is scheduled to run
daily, these individual audits do not need
to be scheduled separately. However, see
Scheduling Notes for RCDCNTALL.
The source for this program can be found
in source file ITHA/E2.CUST. If you
choose to customize this program then
copy the source member to source file
ITHAxx/E2.CUST and compile the
program into ITHAxx.
Special configuration is required for the
CHKOBJMTCH audit in environments
with multiple target nodes. Contact
Check Object Match Daily on primary CALL PGM(E21306CPS) PARM('*ALL' E2xxAOBJMT This audit is not required to be scheduled
CHKOBJMTCH (see Scheduling '*ALL' '*ALL') separately if the AUDSTREAM audit is
Notes) scheduled to run daily.
Special configuration is required in
environments with multiple target nodes.
Contact CustomerCare for details.
Constraint Audit Daily on primary CALL PGM(E21309CPS) PARM(*ALL E2xxACSTP

CST_AUDP *ALL)
Database Relations Weekly on primary CALL PGM(E23390CPS) E2xxAUDDBR

Audit PARM('*ALL')
DBR_AUDIT
Device Configuration Daily on primary if CALL PGM(E23437RP) E2xxADEVP

Audit replicating Device
DEV_AUDP Configuration
Directory Entry Daily on primary if CALL PGM(E21470RP1) PARM('T’ E2xxADIREP

Replication Audit replicating ‘*ALL’ ‘*ALL’)
DIRE_AUDP Directory Entries
Existence Audit - Weekly on primary CALL PGM(E23602CPS) E2xxDEVEX

Device Configuration if replicating
EXIST_DEVP Device
Configuration
Existence Audit - Weekly on primary CALL PGM(E23605CPS) E2xxDIREX

Directory Entries if replicating
EXIST_DIRP Directory Entries
Existence Audit - Job Weekly on primary CALL PGM(E23607CPS) E2xxJBSEX

Schedule Entries if replicating Job
EXIST_JBSP Schedule Entries
Existence Audit - Monthly on CALL PGM(E23604CPS) PARM(*ALL E2xxOBJEX

Object Existence primary *ALL *ALL)
EXIST_OBJP

Name of
Scheduler
Existence Audit - Weekly on primary CALL PGM(E23606CPS) E2xxOTQEX

Output Queues if replicating
EXIST_OUTQ Output Queues
Existence Audit - User Weekly on primary CALL PGM(E23603CPS) E2xxUSREX

Profiles if replicating User
EXIST_USRP Profiles
IFS Audit Daily on primary if CALL PGM(E27590CP1) E2xxAUDIFS

IFS_AUDIT replicating IFS
Installation Audit CALL PGM(E25534CPS) E2xxINSTAL

INSTAL_AUD
Job Scheduler Daily on primary if CALL PGM(E21436RA) E2xxAJBSEP

Replication Audit replicating Job
JOBSCDE_P Scheduler
Journal to Object Audit Weekly on primary CALL PGM(E21390CPS) E2xxAJROJR This audit is not required to be scheduled
JRNOBJJRN (see Scheduling separately if the AUDSTREAM audit is
Notes) scheduled to run daily.
Journaled Objects List Daily on primary CALL PGM(E21350CPS) E2xxAJRNOB This audit is not required to be scheduled
Audit (see Scheduling separately if the AUDSTREAM audit is
JRNOBJLST Notes) scheduled to run daily.
Logical File Attribute Every other day on CALL PGM(E21540CPS) PARM(*ALL E2xxAUDLFP
Audit primary *ALL)
LF_AUDP
Library Attributes Audit Daily on primary CALL PGM(E20930CP) PARM('*ALL') E2xxALIBP

LIB_AUDP
Object Attribute Audit Daily on primary E2RUNOAA OBJ(*ALL/*ALL) E2xxOAA

OBJATRAUD OBJTYPE(*ALL)
AUDMODE(*RPTANDFIX)
Ownership and Daily on primary E2AUDAUT TOSYS(*ALL) LIB(*ALL) E2xxAOBAUT

Authorities Audit OBJ(*ALL) TYP(*ALL)
OWNER_AUTH
Record Count – All Weekly on primary CALL PGM(E21200CP) E2xxARCDAL This audit should be scheduled to run
Objects Audit (see Scheduling after the AUDSTREAM audit has
RCDCNTALL Notes) completed.
Record Count – Daily on primary CALL PGM(E21210CPS) E2xxARCDCG This audit is not required to be scheduled
Changed Objects Audit (see Scheduling separately if the AUDSTREAM audit is
RCDCNTCHG Notes) scheduled to run daily.
Spool File Replication Every other day on CALL PGM(E27131RP) PARM('*ALL' E2xxASPLFP
Audit primary if ‘*ALL)
SPLF_AUDP replicating Spool
Files
Source File Attribute Daily on primary CALL PGM(E21541CPS) PARM('*ALL' E2xxASRCP

Audit '*ALL')
SRC_AUDP
Stored Procedures Audit Daily on primary CALL PGM(E21316CPS) PARM(*FIX E2xxASPROC

STOR_PRC_P *ALL *ALL)
System Values Audit Daily on primary CALL PGM(E26610RP1) PARM(*ALL) E2xxASYSVP

SYSV_AUDP
Triggers Audit Daily on primary CALL PGM(E21307CPS) E2xxATRGP

TRG_AUDP

Name of
Scheduler
User Profile Audit Weekly on primary CALL PGM(E26050CPS) HAUSRAUD

USRPRF_AUD
V6R1 Conversion Monthly on all CALL PGM(E27050CPS) E2xxAV6R1 This audit should run monthly until all
Audit nodes (see PARM(*ALLUSR N) objects that would prevent successful
V6R1_AUDIT Scheduling Notes) upgrade to i5/OS V6R1 are resolved.
Other Scheduled Audits

The following routine jobs should be scheduled as indicated.
Name of Job
Description Required? Job Schedule Entry in Job Scheduling Notes
Scheduler
iTERA HA Weekly on ADDJOBSCDE E2SYNCNML

Non-Mirrored primary JOB(E2SYNCNML)
Library Sync CMD(SYNCNMLIB
E2SYNCNML LIB(*ALL) SYS(*ALL))
FRQ(*WEEKLY)
SCDDATE(*NONE)
SCDDAY(*SUN)
SCDTIME('17:00:00')
USER(HAA1ADMIN)
TEXT(‘iTERA HA Non
Mirrored Library Sync.')

Name of Job
Description Required? Job Schedule Entry in Job Scheduling Notes
Scheduler
IFS Audit Purge Daily on ADDJOBSCDE E2IFSPRGA

History target JOB(E2IFSPRGA)
E2IFSPRGA CMD(E2IFSPRGA
KEEPDAYS(4) KEEPFAIL(7))
FRQ(*WEEKLY)
SCDDATE(*NONE)
SCDDAY(*SUN)
SCDTIME('17:00:00')
USER(HAA1ADMIN)
TEXT(‘IFS Audit History
Purge.')
iTERA HA Daily Daily on all ADDJOBSCDE E2CLRSNCOB If doing problem

Cleanup—Data nodes (see JOB(E2CLRSNCOB) determination, set PDAYS
Queues, Save Files, scheduling CMD(E2CLRSNCOB higher.
& Heal Records notes) QDAYS(5) SDAYS(3) Even though on the primary
Purge PRGHEAL(Y) PDAYS(0)) there are no data queues to
E2CLRSNCOB FRQ(*WEEKLY) delete or heal records to
SCDDATE(*NONE) purge, by setting up the
SCDDAY(*ALL) command exactly the same
SCDTIME('17:00:00') on both nodes, there will be
USER(HAA1ADMIN) nothing you need to change
TEXT(‘iTERA HA Daily when you perform a role
Cleanup.') swap.
Modify retention days based
on preference.


IBM OS Upgrade Guidelines,
Instructions, and Compatibility B
Introduction
This appendix contains information about upgrading the operating system
of systems running iTERA HA.
iTERA HA OS Compatibility
The following IBM OS versions are compatible with iTERA HA v6.0:
• V5R3, V5R3M5
• V5R4, V5R4M5
• V6R1, V6R4M5
• V7R1
Guidelines for upgrading IBM OS in relation to iTERA HA

The following guidelines will aid iTERA HA users in a smooth transition in
upgrading to the IBM OS.
Minimum IBM OS Levels

The following table indicates the IBM OS releases, when they were made
generally available, and when support has ended (or when it will end). As of
this writing, the following OS versions are currently supported by IBM:

Appendix B: IBM OS Upgrade Guidelines, Instructions, and Compatibility
For up-to-date OS supported version information, refer to the following link

to IBM’s website:
http://www-947.ibm.com/systems/support/i/planning/software/i5osschedule.
html
NOTE
iTERA HA v6.0 will run on V5R3 or higher only.
IBM Cumulative Fix Package and PTF Groups

Vision requires that all systems that have iTERA HA installed have the latest
Cumulative Fix Package, Hiper, and Database Group installed. Refer to IBM’s
Recommended Fixes page for more information:
http://www-912.ibm.com/s_dir/slkbase.nsf/recommendedfixes
The following PTF Groups are required:
• Hiper
• DB2 UDB for System i
• TCP/IP Group PTF
• Java
• Backup Recovery Solutions
These groups should be installed on all systems in the cluster, and must be at
the same level on all systems. Notify CustomerCare if your systems run
different OS versions.
For more information, see:
http://www-912.ibm.com/s_dir/sline003.nsf/GroupPTFs
IBM recommends permanently applying all licensed program PTFs prior to

performing an OS upgrade.
IBM PTFs
Consult the document IBM Required PTFs [Published Date].pdf, available from
Support Central, for the list of IBM PTFs that are required for your iTERA
HA installation.

iTERA PTFs
iTERA PTFs
Permanently apply all iTERA HA, iTERA Alert, and Cross Product PTFs
prior to upgrading the OS.
IMPORTANT
Do not permanently apply iTERA HA PTFs via an IPL. Permanently
applying iTERA PTFs during an IPL may result in the IPL hanging
because the iTERA libraries will not be in the library list.
IMPORTANT
All iTERA HA, iTERA Alert, and Cross Product PTFs must be
applied permanently prior to upgrading. Failure to do so could result
in losing some or all of the PTFs.
To permanently apply iTERA PTFs see “Important Information: Permanently

Applying PTFs” on page 244.
Running the Primary and Target nodes on different levels of

the IBM OS
IMPORTANT
Vision Solutions STRONGLY recommends that all nodes run the
same OS level.
IBM tries to support backward compatibility. However, Vision

Solutions has found that they are not always able to achieve this goal.
For example, we know that IBM does not fully support SQL updates
being converted from V5R4 or V5R3 to lower levels. SQL updates
done on V5R4 do not always get journaled in a way that V5R3 can
retrieve the data. This results in the apply process not processing the
update. This is just one of the reasons we do not support the primary
being at a higher release than the target.
iTERA HA can support running different OS versions, for a limited time, as

long as the target node is running the higher OS level (e.g., primary on V5R4 and
target on V6R1 or primary on V6R1 and target on V7R1). The lower
operating system must be upgraded prior to (or during) the next role swap.
iTERA does not support replication for more than one version level difference
(e.g., V5R4 to V7R1).

IMPORTANT
iTERA HA does not support the primary node running a higher OS
level. Additionally, errors may be encountered when running
different OS versions even if the primary version is lower than the
target.
IF you must run on different OS versions, adhere to the following:
• The target node must be running the higher level operating system.
• All objects must be compiled on the primary node using the lower level
operating system. (For example, the primary node could run release V5R3
and the target node run release V5R4.) When the primary node is using
the lower level of the operating system it guarantees that this will be done.
• You cannot perform a role swap to the backup node until you are ready to
upgrade the primary node to the higher OS level. However, you can
perform a failover in the case of the primary node going down.
When ready to upgrade the OS on the primary node:
1. Perform a role swap using the iTERA HA Role Swap Checklist (see “Role
Swap Procedure” on page 205) and then perform all instructions in the
“Pre-OS Upgrade Instructions” on page 273 and “Post-Upgrade
Instructions” on page 273. (If needed, an onsite visit from a Vision
Solutions Consultant is available as a billable service.)
NOTE
While a role swap is recommended, there are circumstances that may
preclude performing one in conjunction with an OS upgrade.
However, performing a role swap will result in less downtime.
NOTE
Errors may be encountered during or following the role swap due to
the different OS versions. CustomerCare will assist you in working
through the issues. However, some issues cannot be resolved until the
lower OS machine is upgraded.
2. Ensure that the replication and apply processes are working.
3. End the iTERA HA subsystems on all nodes (primary first, then target).
4. If you performed a role swap, you may allow users to log on to the new
primary node (formerly the backup).
5. Perform the OS upgrade then restart the node. Do not start application
jobs yet.

Steps to perform the OS Upgrade
6. Restart iTERA HA and let the apply process and object replication process
get caught up.
7. Optional: If a role swap was performed, execute another role swap to

return to the original primary.
Steps to perform the OS Upgrade
Pre-OS Upgrade Instructions

Vision Solutions recommends the following method for upgrading the
Operating Systems where those systems are running iTERA HA.
1. Perform all audits to verify the systems are in sync.
2. Perform all steps on the iTERA HA Monitoring Checklist. See

“Monitoring Checklist” on page 161. Part of monitoring includes
reviewing the System Monitor and the Role Swap Readiness Monitor.
3. Suspend replication by ending the iTERA HA subsystem on all nodes

(primary first, then target).
4. Perform the OS upgrade on the designated node.
Post-Upgrade Instructions
1. On all nodes, use the command CHGDDMTCPA (F4 to prompt) to verify
that the password required field is set to *NO. (For V6R1 and later, this
can be set to *USRID.)
IMPORTANT
If updates are done to the database using SQL, there may be
additional objects requesting sync. This is caused by the primary
running on a newer version of the OS than the target. In general, the
target’s OS should be equal to or newer than the primary. Of course,
this is not possible when upgrading the OS.
2. For V6R1, Vision recommends you perform the IBM conversion for
application programs and service programs for objects that are compiled at
lower OS versions. The program conversion command is STROBJCVN.
Refer to IBM documentation for instructions on using this command.
3. Execute the command CLRPFM E2PAUOBJ on all nodes.

IMPORTANT
This step must be done on all systems after each system has been
upgraded. Failure to do so may result in iTERA resyncing the entire
system.
4. Execute the command CLRPFM on the backup node on the following

files:
CLRPFM E2POAAHD
CLRPFM E2POAALS
CLRPFM E2POAAOD
CLRPFM E2POAAXD
CLRPFM E2POAARM
CLRPFM E2POAARS
CLRPFM E2POAARR
5. On all nodes, access menu option 30.3 from within iTERA HA and delete
all entries beginning with “E”. However, if the *LOCAL entry begins with
“E”, do not delete it. (These entries will be automatically rebuilt when the
subsystem is restarted.)
6. Resume replication by starting the iTERA HA subsystem on all nodes

(target first, then primary).
7. Execute the Check Object Match audit on the primary node (1.8, option
8=Submit on CHKOBJMTCH).
8. Check 1.1, F6=Objects Requesting Sync on the backup. Ensure all is
synced.
9. After replication is caught up (verify in the System Monitor, 1.1), verify
the systems are in sync by executing all audits and the iTERA HA
Monitoring Checklist. See “Monitoring Checklist” on page 161.
10. When all systems are running the same OS version, another role swap may
be performed, if needed.

Application/Software Upgrades
Affecting Mirrored Libraries C
This procedure should be used when performing in-house or third-party

software upgrades affecting mirrored libraries. This is most important if
major database changes will occur.
Optional: To test the application upgrade process on the target node prior to
“going live” on the primary, follow the procedure through step 9 (the “Do
upgrade as instructed by Vendor” step), upgrade target node only, then
perform tests on the application. After successfully testing the application
upgrade, load it on the primary node and continue with step 10 of this
checklist. Any changes to objects on the target node that have been affected
by testing will be resynced.
NOTE
If there are issues with the upgrade that cannot be resolved through
the vendor within a reasonable amount of time, you can opt to not
upgrade the primary, follow any instructions they provide for
removing the application from the target, then resync any libraries
or objects that have been affected.
IMPORTANT
PLEASE read the application’s upgrade documentation very
carefully before starting the upgrade in order to become familiar
with the process.
Procedure for Software Upgrades affecting Mirrored

Libraries
1. Turn off the Audit Command Console (6, F11, both nodes).
2. Place any audits scheduled in the job scheduler on hold

(WRKJOBSCDE, option 3 on all iTERA HA audits).
3. Place the Journal Manager job on hold (3.33, option 3, on all nodes).

Appendix C: Application/Software Upgrades Affecting Mirrored Libraries
4. On the Primary, select Work with Libraries (4.11). Select option

14=Cancel Syncing on the libraries being upgraded. Press Y to confirm.
5. While still in 4.11, select option 4=End Journaling on the libraries being
upgraded to end journaling on the mirror journals. This submits job
xxJE2JRNJ.
6. Check E2MSGLOG on the primary. An entry will indicate the number of

objects for which journaling was ended and how many locked objects there
were. There should be no locked objects. If there are locked objects,
determine which job has the object locked (WRKOBJLCK), release the
locks, then repeat step 5 again.
7. Do these steps only if there are USER journals affected by the upgrade.
a. On the primary, select Work with Local Journals (3.1). Select option
32=Toggle Mirror Status to set the mirror process to OFF for the
USER journals affected. Select Work with Local Journals (3.1,
Primary).
b. Select option 14=Remove Journal to remove the journal for the USER
journals being affected. Follow the application’s upgrade instructions
for handling user journals.
8. In IFS Replication (5.2, Primary), access the appropriate File System
(option 5), then use option 7=Cancel Mirroring, then option 8=End
Journaling for all IFS directories affected by the upgrade.
9. End the iTERA HA subsystems on all nodes (E2ENDSBS; primary first,

then targets).
10. Do the upgrade as instructed by the vendor.
IMPORTANT
If your vendor requires you to start journaling to user journals,
journaling must be started prior to starting the iTERA HA
subsystems. Failure to do so will cause the iTERA HA subsystems to
start journaling objects to HA journals immediately upon starting
the subsystem.
11. Follow the application’s upgrade instructions for activating the user
journals.
12. Start the iTERA HA subsystems on all nodes (E2STRSBS; target first,
then primary).
13. In the iTERA HA subsystem (E2SBS; primary), place the sync job on
hold: xx_SNC_nnn; where xx is the two-character CRG code and nnn is
the three-character code indicating the target node).

Procedure for Software Upgrades affecting Mirrored Libraries
14. In the Work with Libraries screen (4.11, primary) use option 21=Quick
Net Sync to sync the libraries that had previously had syncing cancelled on
them (see step 4).
NOTE
Select only one or two libraries at a time in order to avoid network
congestion.
15. In IFS Replication (5.2, primary), use option 4 to quick sync any IFS
directories that were affected by the upgrade.
NOTE
Select only one or two libraries at a time in order to avoid network
congestion.
16. After all libraries have been synced, release the xx_SNC_nnn job (2.11,
option 6=Release, primary).
17. Release the journal manager job (3.33, option 6=Release; all nodes).
18. Verify that replication has caught up by checking the System Monitor (1.1,
F10=Update Monitor, Primary).
19. Turn the Audit Command Console on (1.8, F11=Start Auto Audit, all
nodes)
20. Release any held iTERA HA audits from the job scheduler
(WRKJOBSCDE, option 6=Release).

Appendix C: Application/Software Upgrades Affecting Mirrored Libraries

i5/OS Recommendations
D
This appendix contains guidelines, considerations, recommendations, and

procedures, pertaining to various OS issues in relation to iTERA HA.
System Name Change

Use the following procedure if changing the system name in conjunction
with a role swap, failover, etc. Generally, you should only change the system
name if your applications require it. Most applications will work without
making the change. If you do change the system name use the instructions
below.
IMPORTANT
If you plan on changing the name and/or *LOCAL RDB entry on
more than one node, change one node, cycle the subsystems on all
nodes twice before you change the name and/or *LOCAL entry on
another node.
IMPORTANT
JDBC and ODBC drives do not use the system name. They use
the *LOCAL entry in the relational database. You can check the
RDB entry in 30.3.
NOTE
If using APPC connections you must change LCLCPNAME and
LCLLOCNAME.
If you plan on changing the system name on the current primary to the
system name on the current backup do the following:

Appendix D: i5/OS Recommendations
1. End iTERA HA Subsystems on all nodes (E2ENDSBS - primary first,

then backup).
2. Change the system name on the current backup to an unused name as per
IBM (this is temporary), unless you are rolling back to the original
primary. In that case, use the original backup name.
3. IPL the server (current backup).
4. Check the iTERA HA subsystem on the backup (E2SBS). If active, end it

using the E2ENDSBS command. (The QSTRUP program may have been
modified to start it.)
5. Verify that DDM is active (30.7).
6. On the current backup, sign on using the iTERA HA profile and enter
option 30.21, press enter, then F3 to exit. Select 30.22, press enter, then
F3 to exit. Sign on to the current primary, select option 30.21 and check
to see if the backup system name has changed.
7. Check to see if you are using passwords on DDM files (30.4).
8. Change the current primary system name to the primary system name as
per IBM.
9. IPL the server.

10. Check the iTERA HA subsystem on the primary (E2SBS). If active, end it
using the E2ENDSBS command. (The QSTRUP program may have been
modified to start it.)
11. If using passwords on DDM files, you will need to update the Server
Authority Entries. Use DSPSVRAUTE to display, CHGSVRAUTE to
change.
12. Verify that DDM is active (30.7).
13. On the current primary, sign on using the iTERA HA profile and enter
option 30.21, press enter, then F3 to exit. Select 30.22, press enter, then
F3 to exit. Sign on to the current backup, select option 30.21 and check to
see if the system name has changed.
14. Clear the following files on all nodes:
• CLRPFM(ITHAxx/E2PXMONS)
• CLRPFM(ITHAxx/E2PXAMON)
• CLRPFM(ITHAxx/E2PTIMH)
15. Check TCP/IP Host Table Entries to see if you need to update the system
that has been renamed (CFGTCP, opt 10).
16. Start the iTERA HA subsystem (E2STRSBS - backup; then primary).

Change the *LOCAL RDB Entry
17. Check TCP/IP Domain Information to see if you need to change the host
name (CFGTCP, opt 12).
18. Change the journal receivers for all IFS journals. On the primary, select
3.2, opt 7=Change Journal Receivers.
19. Restart the IFS Apply jobs. On the backup in option 3.4 reset the apply
jobs to the last sequence of the currently attached receiver (11=Suspend,
12=Override, option 2 to use the last sequence number of the currently
attached receiver, F5 to refresh until active, 14=Restart).
Change the *LOCAL RDB Entry

If using ODBC, JDBC, or any other communication protocol, you may need
to change the new primary *LOCAL RDB entry to be the same as it was on
the old primary when executing a role swap.
To change the *LOCAL RDB entry, do the following:

1. Check to see if you are using passwords on DDM files (30.4).
2. Select 30.3 on the appropriate node.
3. Delete *LOCAL (opt 4, opt G).
4. Create *LOCAL on the new primary using the value that was in *LOCAL
on the old primary.
5. Create *LOCAL on all other nodes using a value that is different than the
value used on all other nodes.
6. If you are not using passwords on DDM files, select option 30.4 (Check
DDM Attributes) and verify that the field Lowest Authentication Method
(OS V6R1 and later) or Password Required (earlier releases) is set to *NO.
7. End (E2ENDSBS) and restart (E2STRSBS) the iTERA HA subsystems

twice on all nodes. This will ensure that iTERA HA has the new *LOCAL
name for this node on all the other systems.
8. Do the following if you are replicating IFS or MQ.
a. Change the related journals receivers. From the primary node, use
menu 3.2 option 7=Change Journal Receiver.
b. Reset the apply jobs (for IFS only). On the backup, select menu 3.4 to
reset the apply jobs to the last sequence of the currently attached receiver
(option 11=Suspend Process State, option 12=Override Next Seq #,
option 2 to use the last sequence number of the currently attached
receiver, F5=Refresh until active, option 14=Restart Job).

NOTE
iTERA HA uses SQL CLI in processing data between nodes. The
*LOCAL name is included in the receiver. If you do not change the
receiver and the apply sequence to process the apply jobs for IFS will
not start.
IPL Considerations
When executing an IPL on the primary, in order to prevent communications
issues between primary and target, perform the following steps:
1. End the subsystem on the primary.
2. End the subsystem on the target.

3. Perform the IPL on the primary.
4. After the primary is up and running, start the subsystem on the target first,
then the primary.
Change DDM TCP/IP Attributes (CHGDDMTCPA)

Vision Solutions recommends the DDM TCP/IP attributes be set as follows.
V5R4M5 and lower
If the Password Required parameter is to be set to *YES, or *ENCRYPTED,

verify that the system value QRETSVRSEC is set to ‘1’ then add a server
authorization entry (ADDSVRAUTE) for QDDMSERVER (case-sensitive; all
caps).
• IBM DDM server:

ADDSVRAUTE USRPRF(E2XXADMIN) SERVER(QDDMSERVER)
PASSWORD(PASSWORD)
• Remote system(s):
ADDSVRAUTE USRPRF(E2XXADMIN) SERVER(SYSTEMNAME)
PASSWORD(PASSWORD)
• Remote system node code (specified in iTERA HA menu 30.21):

IBM PTFs
ADDSVRAUTE USRPRF(E2XXADMIN) SERVER(NODECODE)

PASSWORD(PASSWORD)
V6R1M0 and later
NOTE
If you wish use different DDMTCPA options, you will need make
sure that DDM functions between the HA systems.
IBM PTFs
Recommendations for IBM PTFs, PTF Groups, and Cumulative Fix Package
levels are documented on the Recommended Operating System Fixes page on
the Vision Solutions website:
http://portal.visionsolutions.com/RecommendedOSFixes.aspx


Troubleshooting
E
This chapter contains procedures for investigating and resolving various

issues that can occur in iTERA.
Role Swap Readiness Monitor Issue Resolution

The following table indicates steps to resolve some of the common issues
that occur on tests in the Role Swap Readiness Monitor. It is not a definitive
list and other solutions may be used. Make sure that you note the difference
between messages starting with ‘RMT:’, and those not starting with the
prefix. All messages are assumed to be displayed from the primary system.
The list does not include resolutions of changing the thresholds or ignoring
the tests except under selected conditions.
Message Problem Resolution
APPLYDATA, APPLYIFS, and APPLYTRAN
RMT: Apply *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE)
[ApplyJobName] is not may be on hold. Too on the target.
active many jobs may be in the The subsystem may be down.
*JOBQ E2SYSJOBQ.
RMT: Apply The job may have been Select option 3.4 on the target to restart all ended jobs.
[ApplyJobName] is not cancelled. Select option R=Restart Job on the journal to restart the
active apply job.
RMT: Apply Get job information such as error message and line
[ApplyJobName] has a number of where the problem occurred. Contact Vision
message Solutions to report problem. You may need to end the
job and restart or remove a record from a transaction
file.
RMT: Apply Release the job on the target.

[ApplyJobName] is held
See also “Troubleshooting the Apply Job” on page 306.

Appendix E: Troubleshooting
AUDIT_MON
CSTIFC (Cluster Interface)
No tests are performed on

the Cluster Interface at
this time.
DDM
See “Troubleshooting DDM” on page 317.
DEV (Device Configuration)
x transaction records The replication program Make sure the xx_DEVREP on the primary is running
found in the file is not running as it and not in MSGW. It may be behind if an audit or
should or is not caught quick sync was just requested.
up.
RMT: x transaction Old entries are found in Clear the file E2PCFEN. You may need to cancel the
records found in the file the E2PCFEN file on the xx_OBJMON job.
target system. This may
be due to unapplied
transactions prior to a
roll.
x error records found in Errors were left from a Clear the file E2PCFPC on the primary system.
the file. prior role swap.
RMT: x error records On the target, select option 5.4 and analyze and errors.
found in the file. If the errors are old or not a problem, you may delete all
entries by pressing F21.
xx_DEVREP is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
may be on hold. The subsystem may be down.
Too many jobs may be in
the *JOBQ
E2SYSJOBQ.
xx_DEVREP is not active The job may have been Select option 2.20 to restart all ended jobs.
cancelled
xx_DEVREP has a Get job information such as error message and line
message number of where the problem occurred. Contact Vision
Solutions to report problem. You may need to end the
file.

xx_DEVREP is held Release the job.
ZM_xxTJRx is not active may be on hold. on the target.
Too many jobs may be in The subsystem may be down.
the *JOBQ
E2SYSJOBQ.
ZM_xxTJRA is not active cancelled. Select option R=Restart Job on the journal to restart the
apply job.
ZM_xxTJRx has a number of where the problem occurred. Contact Vision
file.

ZM_xxTJRx is held
DIRE (Directory Entries Replication)
x transaction records The replication program Make sure the xx_DIRREP on the primary is running
up.
RMT: x transaction Old entries are found in You can clear the file E2PDIEN. You may need to
records found in the file the E2PDIEN file on the cancel the xx_OBJMON job.
be due to unapplied
roll.
x error records found in Errors were left from a Clear the file E2PDIPC on the primary system.
the file. prior role swap.
found in the file. If the errors are old or not a problem, you may delete all

Missing exit point for xxx Directory entry additions Auto replication of directory entries is not set up. To set
and changes are not it up:
copied to the target. 1. From 30.23 on all nodes, verify that the Global State
Automatic replication for *DIRE is enabled.
needs to be restarted.
2. Select option 5.7 on primary system.
3. Press F7=Control.
4. Press F7=Add Exit Program.
5. Press F3 to exit. This should also replicate the process
to the target.
6. On the target, select option 5.7.
7. Press F7=Control. Verify that the Exit Program is
Active.
8. Reload directory entry list (5.7 F18) make sure list is
updated.
9. Press F20 to run audit. This will identify missing
entries and make changes as necessary.
xx_DIRREP is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
may be on hold. The subsystem may be down.
Too many jobs may be in
the *JOBQ
E2SYSJOBQ.
xx_DIRREP is not active The job may have been cancelled. Select option 2.20 to
restart all ended jobs.
xx_DIRREP has a Get job information such as error message and line
file.
xx_DIRREP is held Release the job.
ZM_xxTJRx is not active may be on hold. Too on the target.
many jobs may be in the The subsystem may be down.
*JOBQ E2SYSJOBQ.
ZM_xxTJRA is not active cancelled. Select option R=Restart Job on the journal to restart the
apply job.
file.


ZM_xxTJRx is held
ENCRYPTION
Encryption is not Enable encryption

enabled.
FTP
HEAL
IFS (IFS Replication)
xx_IFSMON is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
may be on hold. Too The subsystem may be down.
many jobs may be in the
*JOBQ E2SYSJOBQ.
xx_IFSMON is not active The job may have been cancelled. Select option 2.20 to
xx_IFSMON has a Get job information such as error message and line
file.
xx_IFSMON is held Release the job.
RMT: Apply ZM_xxIJRx *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE)
is not active may be on hold. Too on the target.
*JOBQ E2SYSJOBQ.
RMT: Apply ZM_xxIJRA The job may have been cancelled. Select option 3.4 on
is not active the target to restart all ended jobs. Select option
R=Restart Job on the journal to restart the apply job

RMT: Apply ZM_xxIJRx Get job information such as error message and line
has a message number of where the problem occurred. Contact Vision
file.
RMT: Apply ZM_xxIJRx Release the job on the target.

is held
IPCONWRN
IPCONWRN test to Connection warning on See “Resolve IP Connection Warnings” on page 300
[remote system] failed the IP address for routing notes as well as detailed instructions on
fixing IP address connection warnings.
ITALERT (iTERA Alert)
iTERA Alert menu Enable it in the Replication Options screen (30.23).

options are not available.
iTERA Alert is not set up Refer to iTERA Alert configuration instructions in the
correctly. iTERA HA v6.0 Advanced Features Guide.
E2A_NOTIFY is in jobq Release the job.
E2A_NOTIFY is not Restart the job.

active
E2A_NOTIFY is held Release the job.
E2A_EMAIL is in jobq Release the job.
E2A_EMAIL is not active Restart the job.
E2A_EMAIL is held Release the job.
JOBQ
JOBSCDE (Job Scheduler Replication)
x transaction records The replication program Make sure the xx_JBSREP on the primary is running
up.

RMT: x transaction The replication program Make sure the xx_JBSREP on the target is running and
records found in the file is not running as it not in MSGW. It may be behind if an audit or quick
should or is not caught sync was just requested.
up.
x error records found in Errors were left from a Clear the file E2PJBSF on the primary system.
the file prior role swap.
found in the file If the errors are old or not a problem, you may delete all
xx_JBSREP is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ
xx_JBSREP is not active The job may have been cancelled. Select option 2.20 to
xx_JBSREP has a message Get job information such as error message and line
number of where the problem occurred. Contact Vision
Solutions to report the problem. You may need to end
the job and restart or remove a record from a
transaction file.
xx_JBSREP is held Release the job
Object Go into job schedule screen on both systems by

QUSRSYS/QDFTJOBS selecting option 5.5 on both the primary and target.
CD *JOBSCD has the
wrong Audit Level
*JOBQ E2SYSJOBQ.
RMT: Apply The job may have been cancelled. Select option 3.4 on
ZM_xxTJRA is not active the target to restart all ended jobs. Select option
file.

ZM_xxTJRx is held

MQ (WebSphere MQ Replication)
Library QMQM does not Load library QMQM or turn off *MQ replication in
exist 30.23.
OBJMON1 (Object Monitor 1)
xx_OBJMON is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ.
xx_OBJMON is not The job may have been Select option 2.20 to restart all ended jobs.
active cancelled.
xx_OBJMON has a Get job information such as error message and line
file.
xx_OBJMON is held Release the job
x transaction records The replication program Make sure the OBJMON2 job on the primary is
found in the file is not running as it running and not in MSGW. It may be behind if an
should or is not caught audit or quick sync was just requested.
up.
RMT: x transaction Old entries are found in Clear the file E2PZCEN on the target. You may need to
records found in the file the E2PZCEN file on the cancel the OBJMON2 job.
be due to unapplied
roll. There was an old
problem where library
changes (or object
deletions) was creating
entries on the target file.
OBJMON2 job is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ.
OBJMON2 job is not The job may have been cancelled. Select option 2.20 to
active restart all ended jobs.

OBJMON2 job has a Get job information such as error message and line
file.
OBJMON2 job is held Release the job
*JOBQ E2SYSJOBQ.
ZM_xxTJRA is not active the target to restart all ended jobs.
file.

ZM_xxTJRx is held
OBJSNCSTS (Object Sync Status)
OS (Objects Sync Request)
RMT: x transaction Make sure the xx_SNC_xx job on the primary is

records found in the file running and not in MSGW. It may be behind,
especially during an initial sync, after an audit, or after
many objects have been reorganized.
Review the file E2POSR on the target to verify that data
is replicating as it should. You can also view the data by
taking option 1.1 F6 on the primary.
xx_SNC_xx is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ.

xx_SNC_xx is not active The job may have been cancelled. Select option 2.20 to
xx_SNC_xx has a Get job information such as error message and line
file.
xx_SNC_xx is held Release the job
PING
RCDCNT (Record Count Audit Test)
RJSTS
RMTCMD (Remote Commands (*)
Transaction records found Resolution 1:

in the file Make sure the xx_RMTCMD job on the primary is
OR running and not in MSGW. It may be behind,
x transaction records especially during an initial sync, after an audit, or after
found in the file many objects have been reorganized.
Review the file E2PRMTCMD on the selected system
to verify that data is replicating as it should.
Resolution 2:
Clear the file E2PRMTCMD. Be aware that if you
perform this then you should run all audits as you may
have cancelled replication of one or more objects.
xx_RMTCMD is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ.
xx_RMTCMD is not The job may have been cancelled. Select option 2.20 to

xx_RMTCMD has a Get job information such as error message and line
file.
xx_RMTCMD is held Release the job.
SPLF (Spool File Replication)
RMT: x transaction Make sure the xx_RPTREP on the primary is running

records found in the file and not in MSGW. It may be behind if an audit or
quick sync was just requested.
x error records found in Errors were left from a Clear the file E2PCFPC on the primary system.
the file prior role swap
RMT: x error records

found in the file
xx_RPTREP is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ.
xx_RPTREP is not active The job may have been cancelled. Select option 2.20 to
xx_RPTREP has a Get job information such as error message and line
file.
xx_RPTREP is held Release the job.
xx_RPTREP is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ.
RMT: xx_RPTCHG is The job may have been cancelled. Select option 2.20 on
not active the target to restart all ended jobs.
RMT: xx_RPTCHG has Get job information such as error message and line
a message number of where the problem occurred. Contact Vision
file.
RMT:xx_RPTCHG is Release the job on the target.

held

*JOBQ E2SYSJOBQ.
ZM_xxTJRA is not active the target to restart all ended jobs. Select option
file.

ZM_xxTJRx is held

x transaction records Old entries are found in Resolution 1:

found in the file the E2PRPEN file on the Make sure the xx_RPTREP on the primary is running
target system. and not in MSGW. It may be behind if an audit or
quick sync was just requested.
Resolution 2:
If there is a delay on adds then the number of reports
threshold may need to be increased. Select option 2 on
the *SPLF record and up the transaction threshold to a
higher number.
Resolution 3 (For O/S V5R4 or later):
There are several compression methods available. The
default is typically *MEDIUM. The reason is this
method typically compresses a report by 90%. This
dramatically reduces the communication bandwidth. It
does Select about 3 times longer to save the report and
therefore can slow down the replication of reports.
Users that have a lot of reports and where bandwidth is
less of a concern have switched the compression to
*LOW or *YES with good results.
If bandwidth is not a concern, you can change the
compression to *LOW (*YES) or *NO. There two
methods of changing the value. One affects only spool
files. The other affects object replication.
Alternative 1 (spool file only)
1. Select option 5.3 on the primary system.
2. Press F20=Control. Enter *LOW (or *NO if desired)
in the Compression Method field.
4. Restart job xx_RPTREP on the primary system.
Alternative 2 (also affects other replication)
1. Select option 10.50 on the primary system.
2. Position to E2SAVCMP parameter, press enter
3. Enter a 2 in front of the E2SAVCMP. Enter *LOW
(or *NO if desired) in the Save compression method
used.
4. Restart job xx_RPTREP on the primary system.
SQL (Remote SQL Test)

SYSMON (System Monitor)
xx_SYSMON is in jobq *JOBQ E2SYSJOBQ You may need to add more active jobs (ADDJOBQE).
*JOBQ E2SYSJOBQ.
xx_SYSMON is not The job may have been cancelled. Select option 2.20 to
xx_SYSMON has a Get job information such as error message and line
file.
xx_SYSMON is held Release the job.
SYSVAL (System Values)
Value xxx is missing. 1. Using WRKSYSVAL change the system values.

2. Often you will need to perform audits to get the
system back in sync.
3. After you have correctly set the system values,
perform additional processing below.
QAUDCTL = QAUDJRN is no longer 1. For each of the applications, reload the list (typically
*NOQTEMP receiving changes. IBM F18). 4.11 Library Maint, 5.3 Outq and Spool File
or will remove these values Replication, 5.5 Job Scheduler Replication, 5.4
under certain Configuration Replication.
QAUDCTL = *OBJAUD circumstances to keep the 2. Run all of the audits.
or system up. However, it
QAUDCTL = *AUDLVL causes grief. All object
adds, changes, and
deletes will not be
recorded. Also job
schedule changes, report
add, change, deletes,
authority changes will
not be replicated to the
target.
QAUDLVL = *DELETE Deleted objects are not Run the object match audit.
deleted on the target.
QAUDLVL = *OBJMGT Object changes are not Run the object match audit.
replicated to the target

QAUDLVL = Authorities may be Load and apply current iTERA HA PTFs.

*SECURITY wrong. Also user profile Run the Ownership and Authorities Audit.
changes by iSeries
Navigator will not If user profiles have been changed via the iSeries
replicate. Navigator then also run the user profile audit.
QAUDLVL = *SPLFDTA Reports may not be 1. Select 5.3 Spool File Replication.
or correct on the target. 2. Take option F18 to load any missing OUTQs.
QAUDLVL = *PRTDTA 3. Perform the spool file audit.
QAUDLVL - *CREATE New objects may not be Perform the object audit.
replicated.
TAPMLB (Tape Management)
Library QBRM does not Resolution 1:

exist Load library QBRM
Resolution 2:
Turn off *TAPMLB replication in the Replication
Components screen (30.23). Enter a G in front of
*TAPMLB row.
USRPRF (User Profile Replication)
Missing exit point for xxx The exit point program is Select option 5.1 on primary system.
not set up for user profile If F9=Start Replication is shown press F9.
replication. Users are not
being copied to the target If F21=End Replication is shown press F21, then press
system. F9.
To replicate any missing user profiles (and all existing)
press F13=Trigger all profiles. (The trigger all profiles
process can take a long time as there is a 3 second delay
between profiles.)

Resolve IP Connection Warnings

This section provides information on IP routing recommendations as well as
instructions for resolving IP connection warnings.
Routing Notes
IP address configuration for use in iTERA replication should adhere to the
following:
• Each IP address defined on the IBM i, in the subnet, needs a Schowler

route.
• Only the main User/Takeover IP address on each system gets the

Duplicate Route Priority changed to 6 or higher.
• Only the HA IP address gets a *HOST route.

• If the IBM i systems are on different networks, (different subnets with a
router in between) then you will probably only need to create the *HOST
route on each system.
Websites
• Read important information about and obtain the IBM Schowler Route
instruction information here.
• To find the network address of a subnet use the Network/Node Calculator

here: http://www.t1shopper.com/tools/calculate/ip-subnet/ (scroll down to
the Network/Node Calculator)
Fixing User/Takeover IP Address Warnings

1. Verify that an IP Connection Warning exists, and that HA traffic is using
the User interface.
a. From the Role Swap Readiness Monitor on the primary, locate the
IPCONWRN test. If the monitor is inactive, press F10=Activate and
then refresh the screen until results for the IPCONWRN test are
returned. You should see an error as shown below.
IPCONWRN IP Connection Warnings ERR -- IPConWRN Test to [system] Failed
Menu option 10.10 on the primary may also be used to check for
connection warnings. If connection warnings exist the following
message is displayed:
Takeover IP Connection Warnings exist - Check Message Log (E2MSGLOG)

b. On the primary, use menu option 30.21 to view the IP addresses that
iTERA should be using for replication.
Cluster Cluster
Node Node Interface Interface Node
Opt Node Id Code Status IP 1 IP 2 State
SYSTEM1 E0CAD62005 ---------- 172.22.10.80 Inactive
SYSTEM2 E0CAD62006 ---------- 172.22.10.79 Inactive
If you also want to check the User/Takeover IP address, press

F7=Resource Groups, then select option 2=Change Set on the CRG
that you are working on to get the proper User IP address.
Main Takeover IP . . . . 172.22.10.15
...
Takeover IP Map . . . . A1_TKO_IP

Local Main IP Status . . Active
After you retrieve this information, press F12 twice to return to the
Cluster and Node Maintenance screen.
c. On the target node, type the command NETSTAT and select option 3.
Review all DRDAs and remote journals in the Remote Port column.
Press F11 twice to verify the IP addresses being used. In the example
below, the DRDA and RMTJOUR jobs should be using 172.22.10.79.
Remote Remote Local Local

Opt Address Port Address Port Type
172.22.10.15 drda 172.22.10.80 52540 *TCP
172.22.10.15 drda 172.22.10.80 53952 *TCP
172.22.10.15 drda 172.22.10.80 54171 *TCP
172.22.10.15 drda 172.22.10.80 56182 *TCP
172.22.10.15 drda 172.22.10.80 64059 *TCP
172.22.10.15 drda 172.22.10.80 64681 *TCP
172.22.10.15 drda 172.22.10.80 65322 *TCP
172.22.10.15 rmtjour > 172.22.10.80 9346 *TCP
2. End the iTERA subsystem and inactivate remote journaling.
a. After verifying that there are connection warnings and the IP addresses
in Netstat are incorrect, end the iTERA HA subsystem on all nodes
using the command E2ENDSBS (primary first, then targets).
b. After the subsystems have ended, select menu option 3.3, Remote
Journal Maintenance on the primary. Use option 14=Inactivate
Remote Journal for each active journal and press Enter. Verify that the
status of each remote journal indicates Inactive in the Remote Send
column.
Mirror Local Remote

Opt Journal Library Type State Journal Send Description
A1JRNA ITHAA1JRN MIRROR On Active Inactive DATA JOURNAL A
A1JRNB ITHAA1JRN MIRROR On Active Inactive Testing
3. Check the IP addresses and add TCP routes.

a. Select menu option 30.2 (Configure TCP/IP Settings) then select

option 1. Press F11 (Display interface status) to show which IP s are
Active or Inactive. Check the IP addresses and write down each IP
address that is in the same IP segment as the Takeover and HA IP
addresses. Also write down the Subnet Mask.
IMPORTANT
If you’re not sure which IP addresses are in the same IP segment, check the
subnet calculator web link above to verify.
Internet Subnet Interface Alias

Opt Address Mask Status Name
127.0.0.1 255.0.0.0 Active LOCALHOST

172.22.10.15 255.255.0.0 Active *NONE
172.22.10.79 255.255.0.0 Active *NONE
192.168.5.1 255.255.255.0 Active *NONE
b. Press F12 to exit the Interface screen. On the command line type
Netstat. Select option 2. You will see some *DIRECT routes for the
interfaces on the User/Takeover and HA IP subnet. Make note of these
*DIRECT routes; they will be removed automatically as you create
Schowler routes.
Route Subnet Next Route

Opt Destination Mask Hop Available
172.22.0.0 255.255.0.0 *DIRECT *YES
172.22.0.0 255.255.0.0 *DIRECT *YES
c. Press F12 to exit the Netstat screen. Select option 2 to view the Work
with TCP/IP Routes screen (30.2, option 2). Check the existing routes
to be sure you are not creating duplicate routes.
Route Subnet Next Preferred

Opt Destination Mask Hop Interface
*DFTROUTE *NONE 172.22.1.1 *NONE
Make note of the Next Hop on the Default Route. This may be used
later if the systems are on separate networks.
d. Select option 1=Add to add a Schowler route for the User/Takeover IP

address (do this on all systems).
• The Route Destination should be the Network.
• The subnet mask is what you wrote down for that interface from
step 3a.
• The Next Hop is the actual User/Takeover IP address (press enter).
• The preferred interface is also the actual User/Takeover IP address.
• Change the Duplicate Route Priority for this route to a 6.

• Press enter. You should see that the route was added.
Add TCP/IP Route (ADDTCPRTE)
Route destination . . . . . . . > '177.22.0.0'

Subnet mask . . . . . . . . . . '255.255.0.0'
Type of service . . . . . . . . *NORMAL *MINDELAY, *MAXTHRPUT...
Next hop . . . . . . . . . . . . '177.22.10.15'
Address prefix length . . . . . 64 1-128, *HOST, *NONE
Preferred binding interface . . 172.22.10.15
Binding line description . . . . Name
Maximum transmission unit . . . *IFC 576-16388, *IFC
Route metric . . . . . . . . . . 1 1-16
Route redistribution . . . . . . *NO *NO, *YES
Duplicate route priority . . . . 6 1-10, *MEDIUM, *HIGH, *LOW
Text 'description' . . . . . . . *BLANK
e. Select option 1 to add a Schowler route for the other IP addresses.

• The Route Destination should be the Network.
• The subnet mask is what you wrote down for that interface from
step 3a.
• The Next Hop is the actual IP address (press enter).
• The preferred interface is also the actual IP address.
• Leave the Duplicate Route Priority for this route at a 5.
• Press enter; you should see that the route was added.

Subnet mask . . . . . . . . . . '255.255.0.0'
Next hop . . . . . . . . . . . . '177.22.10.79'
Route metric . . . . . . . . . . 1 1-16
f. Add a *HOST route for the HA IP interface.

• The Route Destination should be the backup IP address (or the
primary IP address if you are on the backup system).
• The subnet mask is *HOST.
• The Next Hop is the IP address defined in 30.21 for the system you
are on if both systems are on the same network. **If the systems are
NOT on the same network, you will use the Next Hop of the
Default Route you noted in step 3c.**

• The preferred interface is also the IP address defined in 30.21 for

the system you are on.
• Leave the Duplicate Route Priority for this route at a 5.
• Press Enter; you should see that the route was added.

Subnet mask . . . . . . . . . . *HOST
Next hop . . . . . . . . . . . . '177.22.10.79'
Route metric . . . . . . . . . . 1 1-16
The completed routing entries are displayed in the Work with TCP/IP Routes
screen, as follows:
Work with TCP/IP Routes

System: SYSTEM1
1=Add 2=Change 4=Remove 5=Display
Route Subnet Next Preferred

Opt Destination Mask Hop Interface
*DFTROUTE *NONE 172.22.1.1 *NONE

172.22.0.0 255.255.0.0 172.22.10.15 172.22.10.15
172.22.0.0 255.255.0.0 172.22.10.79 172.22.10.79
172.22.10.80 *HOST 172.22.10.79 172.22.10.79
g. On the system you are on, use Netstat with option 2 to verify that the
*DIRECT routes are gone. The *DIRECT routes will be replaced
with routes similar to the image below.
Route Subnet Next Route

Opt Destination Mask Hop Available
172.22.0.0 255.255.0.0 172.22.10.15 *YES
172.22.0.0 255.255.0.0 172.22.10.79 *YES
h. From the backup system, use Netstat with option 3 and select option 4
on any remaining DRDA and RMTJOUR (under the Local Port
column) to close them. (Do NOT close the ones at the top of the list
with a * for Remote Address and a * for Remote Port.)
Remote Remote Local Local

Opt Address Port Address Port Type
172.22.10.15 as-mgtc > 172.22.10.108 37991 *TCP
172.22.10.15 57157 172.22.10.30 drda *TCP
172.22.10.42 11839 172.22.10.108 telnet *TCP
172.22.10.79 drda 172.22.10.108 51847 *TCP
172.22.10.79 drda 172.22.10.108 58623 *TCP

4. Restart the iTERA subsystem and verify the new routes are being used.
a. Start the iTERA subsystem on all systems (targets first, then primary).
NOTE
It will take longer than usual to start the subsystem on the target node.
NOTE
Update the System Monitor by selecting option 1.1 on the primary,
then F10=UpdateMonitor. This should activate the remote journals.
(Yes/No values display Yes. If not, press F10 again. If, after pressing
F10 three times the remote journals still will not start, select
F16=Process Monitor, select option 5=WRKJRNA for each journal,
press Enter, F16=Work with remote journal information, then
option 13=Activate.)
Role Swap Readiness . . . . . . . OK Error
Local/Remote Journals Active . . . Yes / Yes Yes

Apply Jobs Active . . . . . . . . Yes
Network/Subsystem Active . . . . . Yes / Yes
Journal Entries Not Applied . . . 2

Current Max Apply Latency . . . . :00
Current Max Network Exposure . . . 24:46
24-Hour Max Network Exposure . . . 24:46
Object Requesting Sync . . . . . . None
b. On the backup node, use Netstat and select option 3. Scroll down to
the DRDAs and remote journals. Press F11 twice to verify the IP
addresses being used. Notice how the IP addresses below match what is
shown in the 30.21 screen shot.
172.22.10.79 20706 172.22.10.80 rmtjour > *TCP

172.22.10.79 21608 172.22.10.80 drda *TCP
172.22.10.79 26966 172.22.10.80 ftp-con > *TCP
172.22.10.79 28451 172.22.10.80 ftp-con > *TCP
172.22.10.79 30130 172.22.10.80 drda *TCP
172.22.10.79 30776 172.22.10.80 drda *TCP
172.22.10.79 32389 172.22.10.80 rmtjour > *TCP
172.22.10.79 39237 172.22.10.80 ftp-con > *TCP
172.22.10.79 39681 172.22.10.80 drda *TCP
172.22.10.79 40893 172.22.10.80 drda *TCP
172.22.10.79 43172 172.22.10.80 drda *TCP
172.22.10.79 45610 172.22.10.80 rmtjour > *TCP
c. On the primary, access the Role Swap Readiness Monitor (1.7) and
locate the IPCONWRN test. If the test is still in WRN status, select
option 7=Run Test.
Sts Sts
Opt Test Description Pri Tgt Results
IPCONWRN IP Connection Warnings OK N/A

d. You can also use menu option 10.10 on the primary to check for
connection warnings. If no warnings exist, the following is displayed:
No Takeover IP connection warnings were found.
5. If connection warnings are still present, do the following.
a. Check the routes to make sure they are correct.

b. Check Netstat with option 3 to be sure that the correct IP addresses are
being used.
c. Check the RDB entries (30.3) to be sure that the correct IP addresses
are defined. (If wrong, fix them by ending the subsystems on all
systems, 3.3 and 14=Inactivate Remote Journal, netstat with option 3
to be sure DRDA and rmtjour is ended, then restart subsystem and
activate remote journaling.)
d. Check the DDM files (WRKDDMF E*) to be sure that only the
correct IP addresses are there. If there are incorrect IP addresses, select
option 4 on all of the DDM files to delete them. Then recreate them
using the E2CRTDDMF command. (If wrong, fix it by ending the
subsystems on all systems, 3.3 and 14=Inactivate Remote Journal,
netstat with option 3 to be sure DRDA and rmtjour is ended, then
restart subsystem and activate remote journaling.)
e. If everything is correct, and netstat 3 is using the proper interfaces and
you’re still getting IP connection warnings, and you’ve signed out and
signed back in then you may need to end TCP services and restart
them. An IPL will also resolve the issue.
Troubleshooting the Apply Job

Use the following procedure for troubleshooting issues with the apply jobs.
1. Select menu 3.6, Journal Apply Statistics on the target system.
a. Press F10 =Restart Statistics.

b. Select F5=Refresh two or more times.
– Is the Apply Sequence number going up?

– What is the job status? (SDQ indicates that a data queue is being
emptied after an object restore.)
– Select option 9=Display Apply Entry.
• If library/object is *UNKNOWN/*UNKNOWN, the entry will
not process. (This issue should be discussed with IBM. See separate
instructions to bypass this entry.)
• If the apply entry will not display, there may be something else
wrong with it. You may have to resend the receiver from primary.
2. Select 3.4 Work with Apply Jobs on target system.
• What is the Process State?

– The status *SEQ MAP can occur after a failover, role swap, virtual
role swap, or other special activity. Suspend, activate, and restart the
apply job. If the apply job does not start processing normally,
contact CustomerCare.
• What is the Job Status?
– If the apply job not is not active, try Suspend, Activate and Restart
on the apply job.
– If Process State goes from Suspend to Active, but Job Sts stays ---,
execute the command WRKJOBQ E2*
• Are there jobs in the E2JOBQ?
– Is the sbs/jobq running the maximum jobs defined?
– Is the E2JOBQ on hold?
• Check E2SBS
– Is the subsystem running the maximum jobs it is allowed?
– Are any jobs in MSGW?
– Are any in DLYW?

Status Indicator
DLY-10 seq # is not found
A message is posted to the joblog if CPF9801 is

received. There are retries for CPF9801 (85
DLY-85
second time out). A message is also posted to the
E2MSGLOG.
DLY-99
DDM issue, may be SQL CLI related. IFS apply

TIMW jobs use SQL CLI. Verify the RDB entries are
correct on all systems.
• What is the Receiver value?

• What is the Next Seq# to Process?
3. Select 3.2 Journal Receiver Maintenance on the target system.
a. Select option 5=WRKJRNA.

b. Select F15=Work with Receiver Directory.
• Are there any partial receivers? Partial receivers cannot be processed and
will stall the apply job. These need to be deleted. In most cases, when

there is a partial receiver in the chain, all the receivers in the chain will
need to be deleted.
a. Select 1.1 System Monitor
b. Press F10=Update Monitor on the primary to send the receivers
across.
• Is the receiver the apply job needs present? If not, do the following:
a. Select 3.2 Journal Receiver Maintenance on the primary system.
b. Select 5=WRKJRNA.
c. Select F15 Work with Receiver Directory. Is the receiver the apply
job needs present? If not, all the libraries and/or objects for this
journal need to be resynced.
4. Check the subsystem E2SBS on the target system.
a. Select option 5=Work with.

b. Select option 11=Display call stack, if active.
c. Is Cancel Heal Req listed? The apply job will not proceed until Cancel
Heal Req function is completed. This process can be sped up by ending
the Heal job in the primary subsystem.
IMPORTANT
In most cases, resetting the apply job by overriding the sequence
number will require a resync.
Valid reasons to reset the apply job

IMPORTANT
Resetting the apply job should only be done under the direction of
CustomerCare. If not done correctly, data integrity issues could
occur, resulting in the need to resync all affected libraries.
• Initial system sync.
• New journal - no objects replicating. Need to use second journal, first

cannot always be replicated because it will reside in local library on target.
• Application upgrade. The user journals are removed from iTERA HA so

they don’t interfere with the upgrade. After the upgrade is complete, the
journals are added back into iTERA HA and the related libraries are
resynced.

Avoidable situations that can result in resetting

the apply job
• Partial receivers on primary.
– Replicated part of receiver

– Deleted on primary
– Restored partial on primary
– Apply job missing entries
NOTE
Restored receivers do not go back into the chain and cannot be sent
to target via remote journaling.
• Did a save/restore migration of backup system. The apply jobs were active
on the target system after the save was done. The receivers had also been
deleted from the primary. Resync libraries for the related journals.
• Receivers were deleted on the primary and target before they were applied.
• Primary system save/restore upgrade (if done wrong). Objects in the
journal will need to be resynced. Journaling and iTERA were started after
production activity had begun.
• The Apply job is too far behind to catch up. This could be because of
planned or unplanned down time. The decision on whether to execute the
override should be discussed with CustomerCare and should be based on
the number of entries and the available bandwidth. Objects in the journal
will need to be resynced.
Troubleshooting High Disk Space

The following guidelines will assist you in troubleshooting reasons for high
disk space usage.
Primary System
1. Select the System Monitor
a. Is the % Total Used By Receivers greater than 5%?

b. Are local and remote journals active? If not, start the remote journals.
c. Are the apply jobs active? If not, start the apply jobs.
d. Is there a high number in the Journal Entries Not Applied field?
2. Select F16 Process Monitor.

– Note any Rmt-Jrn Snd Sts of *INACTIVE __________ (potential

communications issue)
– Note Exposed Entries ___________(potential communications issue)
– Note Apply Pending __________(potential runaway primary job)
– Note any apply job Status of SUSPENDED __________ (cannot
apply if suspended)
– Until the receiver entries are applied, the local receiver will not be
deleted.
3. Select 3.2 Work with Journal Receivers
– Note Total in Current Chain for each journal.

– Is the total consistent for each journal? _______
– Are the totals relatively low (below 30) _______
4. Select 3.32 JRM Managed Journals
– Is Mng Rcv Yes or No? _____
5. Select option 8.
– Is the Delete Hours negative?_____

– Is a Save Required? _____
– Is In Use column value YES and an Application listed? _____
– Be sure to check QAUDJRN. It is not included in the System Monitor
% Used by Receivers.
6. Select 3.33 JRM Job Schedule Entry.
– Is the XPJRNMGT job held? _____
7. DSPLIB ITHAxxWRK (iTERA work library containing temporary

objects)
– What is the Number of objects? _____(may not be getting purged)

– When were they created? _____(may not be getting purged)
– What is their Size? _____(may be large object sync save file)
– Incomplete sync attempts will leave save files in WRK library.
8. WRKJOBSCDE JOB(E2*)
– If E2CLRSNCOB is not listed, add it as follows.

Add Job Schedule Entry (ADDJOBSCDE)

Job name . . . . . . . . . . . . > E2CLRSNCOB Name, *JOBD
Command to run . . . . . . . . . > E2CLRSNCOB QDAYS(5) SDAYS(5)
Frequency . . . . . . . . . . . > *WEEKLY *ONCE, *WEEKLY, *MONTHLY
Schedule date . . . . . . . . . > *NONE Date, *CURRENT,
*MONTHSTR...
Schedule day . . . . . . . . . . > *ALL *NONE, *ALL, *MON, *TUE...
+ for more values
Schedule time . . . . . . . . . > 1000 Time, *CURRENT
9. Files that can be Deleted (WRKOBJ filename, 4=Delete)
– These are only used on the target system. They created when needed.
File Description
E2PXMSGLOG Apply job
QDBRPLAY Apply job
10. Files that can be cleared (CLRPFM filename)
– These are only used on a backup system. If there has never been a role
swap or failover these should be empty.
File Description
E2IFSAUDD IFS Audit
E2PAURCV Receiver Audit
E2POSH Object Sync History
E2PSWPL Role Swap Log
E2PZZLG Receiver Audit
E2P5101C Heal
11. xx_OBJMON2 purge
– Program E21526RP will remove old records based upon the value
found in the file definition found in 50.11. The two file types that
allow purges are History and Log. The purge program E21526RP has
one parm. Either *ALL or the file name. It can be run at any time.
12. Files that can be reorganized (RGZPFM filename)
– Files included in xx_OBJMON2 Purge can be reorganized as well as

those purged through other processes. Below are some that commonly

have a large number of deleted records on the primary. iTERA does not
currently have a procedure to reorganize the files.
File Description
E2PAUOAH Ownership and Authority Audit
E2PAUOAL Ownership and Authority Audit
E2PCJRNO Current Journaled Objects
E2PDBRD Database Relations Detail
E2PMSGL Message (can be purged first; 1.1.F11)
E2PMSGLH Message History (can be purged first; 1.1.F11)
E2POTEN ZC Audit Changes
13. WRKOUTQ E2*
– Are there a lot of E2 spool files?

• Select option 5=Work with Spool Files
– Are any of the spool files large?
14. IBM Disk Analysis Report (taken from KB)

– Since it isn’t always apparent what takes up disk space, the instructions
for creating the IBM Disk Analysis Report are provided below.
• To build the Disk Analysis file, use the following command:
SBMJOB CMD(RTVDSKINF) JOB(RTVDSKINF) JOBQ(QSYSNOMAX)
• You can monitor the job by issuing the following:

WRKACTJOB SBS(QSYSWRK)
– Job name = RTVDSKINF

• Once it finishes running, create the print report:
SBMJOB CMD(PRTDSKINF RPTTYPE(*LIB)) JOB(PRTDSKINF)
JOBQ(QSYSNOMAX)
Target System
Perform these steps on the target system.
1. Select 1.1 System Monitor. Check the following for the target node:
– What is the % Total Used By Receivers: _________

– Is this greater than 5%?

– Are the apply jobs active? _____ (start apply jobs)
– Note Journal Entries Not Applied: _________ (not always accurate)
• Is this a high number?
2. Select F16 Process Monitor.

– For each journal:
• What is Apply Pending __________
• Is the apply job Status SUSPENDED __________ (cannot apply if
suspended)
NOTE
The remote receivers are not eligible for deletion by the iTERA
Journal Manager until the receiver entries have been applied.
3. Select 3.6 Journal Apply Statistics.
– For each apply job:

• What is the Job Status?
• What is the Unapplied Entries?
• What is the Average Apply Per/Sec?
• Is the Unapplied Entries going down?
• What is the Estimated Apply Latency?
4. Select 3.7 Heal Status
– What is the number of records needing to be healed? ______
5. Select 3.2 Work with Journal Receivers.
– Note Total in Current Chain for each journal.

• Is the total consistent for each journal? _______
• Are the totals relatively low (below 30) _______
• If high, select W, F15.
• Are there any partial receivers? _____
– Select F7 to Toggle Local/Remote.
• Note Total in Current Chain for each journal.
• Is the total consistent for each journal? _______
• Are the totals relatively low (below 30)? _______

6. Select 3.32 JRM Managed Journals option 8.
– Is the Delete Hours negative?_____

– Is a Save Required? _____
– Is In Use column value YES and an Application listed? _____
– The local receivers on the target should not have an application.
– Be sure to check QAUDJRN. It is not included in the System Monitor
% Used by Receivers.
7. Select 3.33 JRM Job Schedule Entry
– Is the XPJRNMGT job held? _____

8. WRKJOBQ E2*
– Are there a large number of jobs in the job queue? _____

– Does selecting F5 show the number is going down? _____
9. DSPLIB ITHAxxWRK (iTERA work library containing temporary

objects)
– What is the Number of Objects? _____(may not be getting purged)

– When were they created? _____(may not be getting purged)
– What is their Size? _____(may be large object sync save file)
– Incomplete sync attempts will leave data queues and save files in WRK
library.
10. WRKJOBSCDE JOB(E2*)
– If E2CLRSNCOB is not listed, it needs to be added.

– Is E2CLRSNCOB on hold?
– If E2IFSPRGA is not listed and IFS replication is going on, it needs to
be added.
– Is E2IFSPRGA on hold?
11. Files that can be deleted (WRKOBJ filename, 4=Delete)
– These are only used on the target system. They are created when
needed.
File Description
E2PXMSGLOG Apply job
QDBRPLAY Apply job
12. Files that can be cleared (CLRPFM filename)

– These are only used on a primary system. If there has never been a role
swap or failover these should be empty.
File Description
E2PAUOAH Ownership and Authority Audit
E2PAUOAL Ownership and Authority Audit
E2PCJRNO Current Journaled Objects
E2PDBRD Database Relations Detail
E2POTEN ZC Audit Changes
13. xx_OBJMON2 Purge
Program E21526RP will remove old records based upon the value found
in the file definition found in 50.11. The two file types that allow purges
are History and Log. The purge program E21526RP has one parm. Either
*ALL or the file name. It can be run at any time. xx_PRGLOG is
submitted daily at midnight by xx_OBJMON2 Purge.
14. Files that can be reorganized (RGZPFM filename)
Any file included in xx_OBJMON2 Purge can be reorganized, but below

are some that commonly have a large number of deleted records on the
target.
File Description
E2P5101C Heal
E2IFSAUDD IFS Audit
E2PALLG Audit Level Audit Change Log
E2PAURCV Receiver Audit
E2PDIRL Directory Entry Log *
E2PMSGI Ignored Messages Log *
E2PMSGL Message (can be purged first - 1.1.F11)
E2PMSGLH Message History (can be purged first -1.1.F11)
E2POSH Object Sync History

Troubleshooting DDM
File Description
E2PSWPL Role Swap Log
E2PZZLG Receiver Audit
15. IBM Disk Analysis Report (taken from KB)
– Since it isn't always apparent what takes up disk space, the instructions
for creating the IBM Disk Analysis Report are provided below.
• To build the Disk Analysis file, use the following command:
SBMJOB CMD(RTVDSKINF) JOB(RTVDSKINF) JOBQ(QSYSNOMAX)
• You can monitor the job by issuing the following:

WRKACTJOB SBS(QSYSWRK)
– Job name = RTVDSKINF

• Once it finishes running, create the print report:
SBMJOB CMD(PRTDSKINF RPTTYPE(*LIB)) JOB(PRTDSKINF)
JOBQ(QSYSNOMAX)
Troubleshooting DDM
Errors that indicate DDM Issues

• CPF4528 is: Cannot Close DDM file
• CPF4128 or CPF4207 appeared during OPEN

• Not able to access record
• Job at a DLY 99 (Many iTERA HA jobs go into DLY 99 when DDM is

down)
• CPF4528 appeared during CLOSE
• CPD3E34 - DDM TCP/IP communications error

• I/O operation was applied to closed file
• E2JOBQ having a large number of jobs
• CPF9155 received by procedure E21043CP

• Not able to allocate objects needed for file E2POBJ
• CPF700D received by procedure
• Cannot establish DDM connection with remote node.

• Cannot open DDM file
• Receiver too small
DDM Troubleshooting – Things to check

The DDM Server is an IBM product. Keep track of all the testing you do and
the results so that you can tell IBM if the tips listed below do not solve the
problem.
If you have tried all of the troubleshooting steps and you still have a problem
with the server, you need to call IBM! Let them know all the tests you have run
and the results so that they understand that you have checked for all the
common problems.
• Sign on using the iTERA HA User Profile (all nodes).
• Enter the iTERA HA password (uppercase only).
• Start the DDM Server (30.5, all nodes).
• Run the Ping, FTP, and DDM Check (30.7) to check to see if all three
work both ways. Results can be reviewed in the iTERA HA message log
(E2MSGLOG).
• Check to see if jobs are running using NETSTAT (is the DDM server up?)
• Check DDM Attributes don’t require a password (30.4 all nodes), the field
“Password required” should say “*NO” unless you have set up iTERA HA
to handle passwords on iTERA HA.
• Check to see if DDM has an exit program attached (DSPNETA) using the
command DSPNETA to view the field DDM request access (if a program
has been attached, make sure that the iTERA HA user profile is defined to
the exit program’s application).
• Check to ensure the Relational Database is defined correctly
(WRKRDBDIRE).
• Check JOBQ QSYSNOMAX to ensure that it is not on hold (WRKJOBQ
QSYSNOMAX).
• Try to access data using a DDM file. Use the command “WRKDDMF
ITE2/*all” to find the first file that is attached to the remote file
ITE2/E2PCNOD. Use the DBU or DFU (UPDDTA) to access the
corresponding DDM file. If the Screen requesting the key information
appears, then the DDM file is working.
• Inactivate the Ethernet Card (CFGTCP, 1, 10) then re-activate the Card
(CFGTCP 1, 9). If this corrects the problem, it indicates there is a possible
hardware failure.

Troubleshooting DDM
• Check to see if you are on the latest IBM OS Cumulative PTF. Check the
IBM Web site for the latest Cumulative PTF.
• Check to see if IBM has any new PTFs. Check the IBM Web site for the
latest PTF.

Other Troubleshooting
Mirroring restarts after ending mirroring jobs

If for some reason you have ended mirroring jobs and you don’t want them to
start again until you are ready then:
1. End the iTERA HA subsystem on the primary node first (2.13), then the
target node (2.13).
2. Perform the function you want to perform.

3. Start the iTERA HA subsystem on the target node first (2.13), then the
primary node (2.13).
Object Audit Results on Target Only Shows

Objects Being Added, but None Deleted
Check the data area E2AUDDLT and ensure that it contains a “Y”.

Itera Ha: User Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Itera Ha: User Guide

Uploaded by

Copyright:

Available Formats

iTERA HA

Copyright Vision Solutions® , Inc. 2003–2011

Purpose of this guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 1—Introduction to iTERA HA High Availability . . . . . . . . . . . 3

Chapter 2—Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

iTERA HA v6.0 User Guide iii

Required Scheduled Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Chapter 3—Understanding And Maintaining The Journaling

Chapter 4—Replication Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

iv iTERA HA v6.0 User Guide

Chapter 5—Non-Library Replication . . . . . . . . . . . . . . . . . . . . . . . . . 93

iTERA HA v6.0 User Guide v

Conversion Utility for Directory Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

vi iTERA HA v6.0 User Guide

Ending WebSphere MQ Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Chapter 6—Monitoring Your System . . . . . . . . . . . . . . . . . . . . . . . . 145

Chapter 7—Monitoring Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

Chapter 8—Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Chapter 9—Role Swap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

iTERA HA v6.0 User Guide vii

Virtual Role Swap with Communications Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

Chapter 10—How to Get and Apply iTERA PTFs. . . . . . . . . . . . . . . 235

viii iTERA HA v6.0 User Guide

Appendix A—iTERA HA Scheduled Audits and Other Jobs . . . . . . 263

Appendix B—IBM OS Upgrade Guidelines, Instructions, and Compati-

Appendix C—Application/Software Upgrades Affecting Mirrored Li-

Appendix D—i5/OS Recommendations. . . . . . . . . . . . . . . . . . . . . . 279

Appendix E—Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

iTERA HA v6.0 User Guide ix

Primary System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310

x iTERA HA v6.0 User Guide

Changes in this guide

iTERA HA v6.0 User Guide 1

• Virtual Role Swap Test

• System Health Check

• System Migrations (hardware replacement)

• Full system resynchronization

• Adding new nodes or CRGs

• Removing nodes or CRGs

• Additional product education

2 iTERA HA v6.0 User Guide

Overview of iTERA HA High Availability Terminology

The process of duplicating an object or library

The iTERA HA process of using journaling to

Similar to synchronization, but it does not include

Also referred to as the source, or production node,

The system where an identical copy of selected

iTERA HA v6.0 User Guide 3

A target node that can be used in a role swap or

A target node that is similar to a backup node in

A Cluster Resource Group is an association of

A Journal which records all changes, adds, and

A record of a change made to an object. Changes

A component of the OS/400 journaling process

A journaling process that sends a copy of journal

An iTERA HA job that runs on the target node

An object on the target node that no longer

The same process as synchronization, but

4 iTERA HA v6.0 User Guide

The process of having a backup node take on the

An abrupt ending of the primary node, due to a

A journal created outside of the iTERA HA

A journal that was created through the iTERA HA