V6.

cover

Front cover

Power Systems for AIX III:

Advanced Administration and
Problem Determination

(Course code AN15)

Course Quick View with Full

TOC
ERC 2.1
Course Quick View with Full TOC

Trademarks
The reader should recognize that the following terms, which appear in the content of this
training document, are official trademarks of IBM or other companies:
IBM® is a registered trademark of International Business Machines Corporation.
The following are trademarks of International Business Machines Corporation in the United
States, or other countries, or both:
AIX 5L™ AIX 6™ AIX®
AS/400® Current® DB2®
DS8000® HACMP™ Initiate®
Initiate® MWAVE® Power Systems™
Power® POWER® PowerVM™
POWER6® POWER7® pSeries®
Redbooks® Redbooks® RS/6000®
System p® Tivoli® Tivoli®
Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the
United States and other countries.
Windows is a trademark of Microsoft Corporation in the United States, other countries, or
both.
UNIX is a registered trademark of The Open Group in the United States and other
countries.
VMware and the VMware "boxes" logo and design, Virtual SMP and VMotion are registered
trademarks or trademarks (the "Marks") of VMware, Inc. in the United States and/or other
jurisdictions.
Other product and service names might be trademarks of IBM or other companies.

August 2011 edition

The information contained in this document has not been submitted to any formal IBM test and is distributed on an “as is” basis without
any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer
responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While
each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will
result elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.

© Copyright International Business Machines Corporation 2009, 2011.

This document may not be reproduced in whole or in part without the prior written permission of IBM.
Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is subject to restrictions
set forth in GSA ADP Schedule Contract with IBM Corp.
V6.0
Course Quick View with Full TOC

TOC Contents
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Instructor course overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Course description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Agenda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii

Unit 1. Advanced AIX administration overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Application outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
Maintenance window tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Effective problem management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-11
Before problems occur . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-14
Before problems occur: A few good commands . . . . . . . . . . . . . . . . . . . . . . . . . . 1-17
Steps in problem resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-19
Progress and reference codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-23
Reference codes at IBM Information Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-26
Working with AIX Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-28
AIX Support test case data (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-31
AIX Support test case data (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-34
AIX software update hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-36
Relevant documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-39
Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-41
Exercise: Advanced AIX administration overview . . . . . . . . . . . . . . . . . . . . . . . . . 1-43
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-45

Unit 2. The Object Data Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
2.1. Introduction to the ODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
What is the ODM? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-6
Data managed by the ODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
ODM components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11
ODM database files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-13
Device configuration summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-16
Configuration manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-18
Location and contents of ODM repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-20
How ODM classes act together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-24
Data not managed by the ODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-26
Let’s review: Device configuration and the ODM . . . . . . . . . . . . . . . . . . . . . . . . . 2-28
ODM commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-30
Changing attribute values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-33
Using odmchange to change attribute values . . . . . . . . . . . 2-36
2.2. ODM database files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-39

© Copyright IBM Corp. 2009, 2011 Contents iii

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
Course Quick View with Full TOC

Software vital product data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-40

Software states you should know about . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-43
Predefined devices (PdDv) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-46
Predefined attributes (PdAt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-51
Customized devices (CuDv) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-54
Customized attributes (CuAt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-58
Additional device object classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-60
ODM and high-level device commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-63
Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-66
Exercise: The Object Data Manager (ODM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-68
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-70

Unit 3. Error monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-2
3.1. Working with the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-5
Error logging components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-6
Generating an error report using SMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-9
The errpt command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-13
A summary report (errpt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-16
A detailed error report (errpt -a) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-18
Types of disk errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-22
LVM error log entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-25
Maintaining the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-28
Exercise: Error monitoring (part 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-31
3.2. Error notification and syslogd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-33
Error notification methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-34
Self-made error notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-37
ODM-based error notification: errnotify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-40
syslogd daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-44
syslogd configuration examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-47
Redirecting syslog messages to error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-51
Directing error log messages to syslogd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-53
System hang detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-55
Configuring shdaemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-58
Checkpoint (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-61
Checkpoint (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-63
Exercise: Error monitoring (part2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-65
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-67

Unit 4. Network Installation Manager basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-2
NIM overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-4
Machine roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-7
Boot process for AIX installation: Tape or CD . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-10
Boot process for AIX installation with NIM (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . .4-13
Boot process for AIX installation with NIM (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . .4-15
NIM objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-17
Listing NIM objects and their attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4-20

iv AIX Advanced Administration © Copyright IBM Corp. 2009, 2011

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

TOC NIM configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23

resources objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-26
resources objects: lpp_source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-29
resources objects: spot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-33
resources objects: mksysb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-37
networks objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-40
machines objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-43
Defining a machine object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-46
Define a client using SMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-49
NIM operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-52
bos_inst operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-57
More information about NIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-60
Additional topics in NIM course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-63
Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-65
Exercise: NIM basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-67
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-69

Unit 5. System initialization: Accessing a boot image . . . . . . . . . . . . . . . . . . . . . . . 5-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
5.1. System startup process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5
How does a System p server or LPAR boot? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-6
Loading of a boot image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-9
Contents of the boot logical volume (hd5) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-12
5.2. Unable to find boot image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-15
Working with bootlists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-16
AIX 7: bootlist pathid enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-19
Starting System Management Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-21
Working with bootlists in SMS (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-24
Working with bootlists in SMS (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-27
5.3. Corrupted boot logical volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-31
Boot device alternatives (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-32
Boot device alternatives (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-35
Accessing a system that will not boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-37
Booting in maintenance mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-41
Working in maintenance mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-44
How to fix a corrupted BLV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-47
Checkpoint (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-50
Checkpoint (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-52
Exercise: System initialization - Accessing a boot image . . . . . . . . . . . . . . . . . . . 5-54
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-56

Unit 6. System initialization: rc.boot and inittab . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-2
6.1. AIX initialization part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-5
System software initialization overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-6
rc.boot 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-9
rc.boot 2 (part 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-12
rc.boot 2 (part 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-15

© Copyright IBM Corp. 2009, 2011 Contents v

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
Course Quick View with Full TOC

rc.boot 3 (part 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-18

rc.boot 3 (part 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-21
rc.boot summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-24
Fixing corrupted file systems and logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-26
Let’s review: rc.boot (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-29
Let’s review: rc.boot (2 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-31
Let’s review: rc.boot (3 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-33
6.2. AIX initialization part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-35
Configuration manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-36
Config_Rules object class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-39
cfgmgr output in the boot log using alog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-42
/etc/inittab file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-45
Boot problem management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-49
Let’s review: /etc/inittab file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-53
Checkpoint (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-58
Checkpoint (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-60
Exercise: System initialization - rc.boot and inittab . . . . . . . . . . . . . . . . . . . . . . . . .6-62
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6-64

Unit 7. LVM metadata and related problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-2
7.1. LVM data representation: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-5
LVM terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-6
LVM identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-9
LVM data on disk control blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-12
LVM data in the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-15
LVM related ODM objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-17
7.2. Export and import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-19
Exporting a volume group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-20
Importing a volume group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-23
importvg and duplicate names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-26
importvg and existing logical volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-29
importvg and existing file systems (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-31
importvg and existing file systems (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-34
7.3. LVM Metadata details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-37
Contents of the VGDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-38
VGDA example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-41
The logical volume control block (LVCB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-46
How LVM interacts with ODM and VGDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-49
ODM entries for physical volumes (1 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-52
ODM entries for physical volumes (2 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-55
ODM entries for physical volumes (3 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-58
ODM entries for physical volumes (4 of 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-61
ODM entries for volume groups (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-63
ODM entries for volume groups (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-65
ODM entries for logical volumes (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-67
ODM entries for logical volumes (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-69
7.4. LVM metadata related problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-71

vi AIX Advanced Administration © Copyright IBM Corp. 2009, 2011

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

TOC ODM-related LVM problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-72

Fixing ODM problems (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-75
Fixing ODM problems (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-78
Intermediate level ODM commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-82
Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-85
Exercise: LVM metadata and related problems . . . . . . . . . . . . . . . . . . . . . . . . . . 7-87
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-89

Unit 8. Disk management procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2
8.1. Failed disks: Mirroring and quorum issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-5
Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-6
Stale partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-9
Mirroring rootvg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-12
VGDA count . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-17
Quorum not available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-19
Nonquorum volume groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-22
Forced vary on (varyonvg -f) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-25
Physical volume states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-28
8.2. Disk replacement techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-31
Disk replacement: Starting point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-32
Procedure 1: Disk mirrored . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-35
Procedure 1: Disk mirrored with replacepv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-38
Procedure 1: Disk mirrored without replacepv . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-40
Procedure 1: Special steps for rootvg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-43
Procedure 2: Disk still working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-45
Procedure 2: Special steps for rootvg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-48
Procedure 3: Disk in missing or removed state . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-51
Procedure 4: Total rootvg failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-54
Procedure 5: Total non-rootvg failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-57
ODM errors from LVM commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-60
Removal of disk without reducevg (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-63
Removal of disk without reducevg (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-65
Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-68
Exercise: Disk management procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-70
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-72

Unit 9. Install and cloning techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-2
9.1. Alternate disk installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-5
Topic 1 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-6
Alternate disk installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-8
Alternate mksysb disk installation (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-11
Alternate mksysb disk installation (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-14
Alternate disk rootvg cloning (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-17
Alternate disk rootvg cloning (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-20
Removing an alternate disk installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-23
NIM alternate disk migration (nimadm) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-26

© Copyright IBM Corp. 2009, 2011 Contents vii

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
Course Quick View with Full TOC

Exercise: Alternate disk install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-29

9.2. Using multibos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-31
Topic 2 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-32
multibos overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-34
Active and standby BOS logical volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-37
Setting up a standby BOS (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-39
Setting up a standby BOS (2 of 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-42
Other multibos operations (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-44
Other multibos operations (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-47
Checkpoint (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-50
Checkpoint (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-52
Exercise: multibos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-54
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9-56

Unit 10. Advanced backup techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-2
Backup data inconsistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-4
Ensuring backup data consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-6
10.1. LVM mirror-based online backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-9
Topic 2 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-10
Online JFS Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-12
Splitting the Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-15
Reintegrate a Mirror Backup Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-18
Snapshot volume groups (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-20
Snapshot volume groups (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-22
Snapshot Volume Group Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-24
Snapshot volume group example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-26
10.2. JFS2 snapshot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-29
Topic 2 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-30
JFS2 snapshot (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-32
JFS2 snapshot (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-34
JFS2 snapshot mechanism (1 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-37
JFS2 snapshot mechanism (2 of 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-39
JFS2 snapshot SMIT menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-41
Creating snapshots (external) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-44
Creating snapshots (internal) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-48
Listing snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-50
Using a JFS2 snapshot to recover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-53
Using a JFS2 external snapshot to back up . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-56
Using a JFS2 internal snapshot to back up . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-59
JFS2 snapshot space management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-62
10.3. SAN Copy issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-65
Topic 3 objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-66
SAN Copy and file system cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-68
Use of JFS2 freeze and thaw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-70
Consistency groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-73
Accessing SAN Copy data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-76
The recreatevg command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10-79

viii AIX Advanced Administration © Copyright IBM Corp. 2009, 2011

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

TOC Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-81

Exercise: Advanced backup techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-83
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-85

Unit 11. Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1

Appendix objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2
When do I need diagnostics? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-4
Where do I run diagnostic? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-7
The diag command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-9
Working with diag (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-11
Working with diag (2 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-14
Working with diag (3 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-17
What happens if a device is busy? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-19
Diagnostic modes (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-21
Diagnostic modes (2 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-24
Diagnostic modes (3 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-27
diag: Using task selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-29
Diagnostic log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-32
Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-34
Exercise: Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-36
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-38

Unit 12. The AIX system dump facility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-1

Unit objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-2
System dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-5
Types of dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-7
How a system dump is invoked . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-10
Crash code: 888 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-13
When a dump occurs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-16
The sysdumpdev command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-18
Firmware assisted dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-23
Dedicated dump device (1 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-26
Dedicated dump device (2 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-28
Dedicated dump device (3 of 3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-31
Estimating dump size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-33
dumpcheck utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-36
Methods of starting a dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-39
Start a dump from a TTY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-43
Generating dumps with SMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-46
Generating dumps with HMC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-49
Dump-related LED codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-51
Copying a system dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-54
Automatically reboot after a crash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-57
Sending a dump to IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-60
Use kdb to analyze a dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-65
Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-69
Exercise: System dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-71
Unit summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-73

© Copyright IBM Corp. 2009, 2011 Contents ix

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
Course Quick View with Full TOC

Appendix A. Checkpoint solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

Appendix B. Command summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1

Appendix C. AIX dump code and progress codes . . . . . . . . . . . . . . . . . . . . . . . . . . C-1

x AIX Advanced Administration © Copyright IBM Corp. 2009, 2011

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

TMK Trademarks
The reader should recognize that the following terms, which appear in the content of this
training document, are official trademarks of IBM or other companies:
IBM® is a registered trademark of International Business Machines Corporation.
The following are trademarks of International Business Machines Corporation in the United
States, or other countries, or both:
AIX 5L™ AIX 6™ AIX®
AS/400® Current® DB2®
DS8000® HACMP™ Initiate®
Initiate® MWAVE® Power Systems™
Power® POWER® PowerVM™
POWER6® POWER7® pSeries®
Redbooks® Redbooks® RS/6000®
System p® Tivoli® Tivoli®
Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the
United States and other countries.
Windows is a trademark of Microsoft Corporation in the United States, other countries, or
both.
UNIX is a registered trademark of The Open Group in the United States and other
countries.
VMware and the VMware "boxes" logo and design, Virtual SMP and VMotion are registered
trademarks or trademarks (the "Marks") of VMware, Inc. in the United States and/or other
jurisdictions.
Other product and service names might be trademarks of IBM or other companies.

© Copyright IBM Corp. 2009, 2011 Trademarks xi

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

pref Course description

Power Systems for AIX III: Advanced Administration and Problem
Determination

Duration: 5 days

Purpose
This course provides advanced AIX system administrator skills with a
focus on availability and problem determination. It provides detailed
knowledge of the ODM database where AIX maintains so much
configuration information. It shows how to monitor for and deal with
AIX problems. There is special focus on dealing with Logical Volume
Manager problems, including procedures for replacing disks. Several
techniques for minimizing the system maintenance window are
covered. While the course includes some AIX 7.1 enhancements,
most of the material is applicable to prior releases of AIX.

Audience
This is an advanced course for AIX system administrators, system
support, and contract support individuals with at least six months of
experience in AIX.

Prerequisites
You should have basic AIX System Administration skills. These skills
include:
• Use of the Hardware Management Console (HMC) to activate a
logical partition running AIX and to access the AIX system console
• Install an AIX operating system from an already configured NIM
server
• Implementation of AIX backup and recovery
• Manage additional software and base operating system updates
• Familiarity with management tools such as SMIT
• Understand how to manage file systems, logical volumes, and
volume groups
• Mastery of the UNIX user interface including use of the vi editor,
command execution, input and output redirection, and the use of
utilities such as grep

© Copyright IBM Corp. 2009, 2011 Course description xiii

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
Course Quick View with Full TOC

These skills could be developed through experience or by formal

training. Recommended training courses to obtain these prerequisite
skills are either of the following:
• Power Systems for AIX II: AIX Implementation and Administration
(AN12) and its prerequisites
If the student has AIX system administration skills, but is not familiar
with the LPAR environment, those skills may be obtained by attending
either of the following:
• AN11 Power Systems Administration I: LPAR Configuration

Objectives
On completion of this course, students should be able to:
• Perform system problem determination and reporting procedures
including analyzing error logs, creating dumps of the system, and
providing needed data to the AIX Support personnel
• Examine and manipulate Object Data Manager databases
• Identify and resolve conflicts between the Logical Volume Manager
(LVM) disk structures and the Object Data Manager (ODM)
• Complete a very basic configuration of Network Installation
Manager to provide network boot support for either system
installation or booting to maintenance mode
• Identify various types of boot and disk failures and perform the
matching recovery procedures
• Implement advanced methods such as alternate disk install,
multibos, and JFS2 snapshots to use a smaller maintenance
window

Contents
• Overview of advanced administration techniques
• Error monitoring
• The Object Data Manager (ODM)
• Basic Network Installation Manager (NIM) configuration
• System initialization problem determination
• Disk management theory and procedures
• Advanced techniques for installation and backup
• The AIX system dump facility

xiv AIX Advanced Administration © Copyright IBM Corp. 2009, 2011

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

pref Agenda
Day 1
Welcome
Unit 1: Advanced AIX administration overview
Exercise 1: Problem diagnostic information
Unit 2: The Object Data Manager
Exercise 2: The Object Data Manager
Unit 3: Error monitoring
(optional) Exercise 2: Part 3

Day 2
Exercise 3: Error monitoring
Unit 4: Network Installation Manager basics
Exercise 4: Basic NIM configuration
Unit 5: System initialization: Accessing a boot image
Exercise 5: System initialization: Accessing a boot image

Day 3
Unit 6: System initialization: rc.boot and inittab
Exercise 6: System initialization: rc.boot and inittab
Unit 7: LVM metadata and related problems
Exercise 7: LVM metadata and related problems
Unit 8: Disk management procedures, topic 1
Exercise 8: Part 1
(optional) Exercise 6: Part 3
(optional) Exercise 7: Part 6

© Copyright IBM Corp. 2009, 2011 Agenda xv

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
Course Quick View with Full TOC

Day 4
Unit 8: Disk Management procedures, topic 2
Exercise 8: Parts 2 and 3: Disk management procedures
Unit 9, topic 1: Alternate disk
Exercise 9, part 1 - Alternate disk
Unit 9, topic 2: multibos
Exercise 9, part 2: multibos
Unit 10, topic 1: LVM mirror-based online backups
Exercise 10, part 1: Snapshot volume group
Unit 10, topic 2: JFS2 snapshot
Exercise 10, parts 3 and 4: JFS2 snapshot

Day 5
Unit 10, topic 3: SAN Copy issues
Unit 11: Diagnostics
Exercise 11: Diagnostics
Unit 12: The AIX system dump facility
Exercise 12: System dump facility
Wrap up / Evaluations

xvi AIX Advanced Administration © Copyright IBM Corp. 2009, 2011

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 1. Advanced AIX administration overview

What this unit is about

This unit introduces various AIX administration issues related to
problem determination and handling system maintenance and backup
in an efficient manner.

What you should be able to do

After completing this unit you should be able to:
• List the steps of a basic methodology for problem determination
• List AIX features that assist in minimizing planned downtime or
shortening the maintenance window
• Explain how to find documentation and other key resources
needed for problem resolution

How you will check your progress

Accountability:
• Checkpoint questions
• Lab exercise

References
SG24-5496 Problem Solving and Troubleshooting in AIX 5L
(Redbook)
SG24-5766 AIX 5L Differences Guide Version 5.3 Edition
(Redbook)
SG24-7559 IBM AIX Version 6.1 Differences Guide (Redbook)

© Copyright IBM Corp. 2009, 2011 Unit 1. Advanced AIX administration overview 1-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 2. The Object Data Manager

What this unit is about

This unit describes the structure of the Object Data Manager (ODM). It
shows the use of the ODM command line interface and explains the
role of the ODM in device configuration. Specific information regarding
the function and content of the most important ODM files is also
presented.

What you should be able to do

After completing this unit, you should be able to:
• Describe the structure of the ODM
• Use the ODM command line interface
• Explain the role of the ODM in device configuration
• Describe the function of the most important ODM files

How you will check your progress

Accountability:
• Checkpoint questions
• Lab exercise

References
Online AIX Version 7.1 Command Reference volumes 1-6
Online AIX Version 7.1 General Programming Concepts:
Writing and Debugging Programs
Online AIX Version 7.1 Technical Reference: Kernel and
Subsystems
Note: References listed as “online” above are available through the
IBM Systems Information Center at the following address:
http://publib.boulder.ibm.com/infocenter/systems

© Copyright IBM Corp. 2009, 2011 Unit 2. The Object Data Manager 2-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 3. Error monitoring

What this unit is about

This unit covers techniques in monitoring for problems and how to
automate responses to those problems. Topics include an overview of
the AIX Error Log facility (and how it can interact with the syslogd
daemon), and the system hang (shdaemon) monitoring facility.

What you should be able to do

After completing this unit, you should be able to:
• Analyze error log entries
• Identify and maintain the error logging components
• Describe different error notification methods
• Log system messages using the syslogd daemon
• Monitor and take actions for hang conditions using shdaemon

How you will check your progress

Accountability:
• Lab exercise
• Checkpoint questions

References
Online AIX Version 7.1 General Programming Concepts:
Writing and Debugging Programs (Chapter 5.
Error-Logging Overview)
Online AIX Version 7.1 Command Reference volumes 1-6
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/eserver

© Copyright IBM Corp. 2009, 2011 Unit 3. Error monitoring 3-1

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 4. Network Installation Manager basics

What this unit is about

This unit provides an introduction to using the Network Installation
Manager (NIM) to network boot an AIX client system. It covers the
basic installation and configuration of NIM for supporting client
installation or booting to maintenance mode.

What you should be able to do

After completing this unit, you should be able to:
• Configure an AIX partition for use as a NIM master
• Set up NIM to support the installation of AIX onto a client

How you will check your progress

Accountability:
• Checkpoint
• Machine exercises

References
SC23-6616 AIX Version 7.1 Installation and migration
SG24-7296 NIM from A to Z in AIX 5L (Redbook)
http://www.redbooks.ibm.com
IBM Redbooks

© Copyright IBM Corp. 2009, 2011 Unit 4. Network Installation Manager basics 4-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 5. System initialization: Accessing a boot

image

What this unit is about

This unit describes the boot process up to the point of loading the boot
logical volume. It describes the content of the boot logical volume and
how it can be recreated, if it is corrupted.

What you should be able to do

After completing this unit, you should be able to:
• Describe the boot process through to the loading of the boot logical
volume
• Describe the contents of the boot logical volume
• Recreate the boot logical volume on a system which is failing to
boot
• Adjust the bootlist for the desired order of search

How you will check your progress

Accountability:
• Checkpoint questions
• Exercise

References
Online AIX Version 7.1 Operating system and device
management
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/systems

SG24-5496 Problem Solving and Troubleshooting in AIX 5L

(Redbook)

© Copyright IBM Corp. 2009, 2011 Unit 5. System initialization: Accessing a boot image 5-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 6. System initialization: rc.boot and inittab

What this unit is about

This unit describes the final stages of the boot process and outlines
how devices are configured for the system.
Common boot errors are described and how they can be analyzed to
fix boot problems.

What you should be able to do

After completing this unit, you should be able to:
• Identify the steps in system initialization from loading the boot
image to boot completion
• Identify how devices are configured during the boot process
• Analyze and solve boot problems

How you will check your progress

Accountability:
• Checkpoint questions
• Lab exercise

References
Online AIX Version 7.1 Operating system and device
management
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/systems

SG24-5496 Problem Solving and Troubleshooting in AIX 5L

(Redbook)

© Copyright IBM Corp. 2009, 2011 Unit 6. System initialization: rc.boot and inittab 6-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 7. LVM metadata and related problems

What this unit is about

This unit explains metadata concepts important for understanding and
working with AIX logical volume manager problems.

What you should be able to do

After completing this unit, you should be able to:
• Explain where LVM metadata information is stored
• Use importvg and exportvg to manage LVM metadata
• Solve ODM-related LVM problems

How you will check your progress

Accountability:
• Checkpoint questions
• Lab exercises

References
Online AIX Version 7.1 Command Reference volumes 1-6
Online AIX Version 7.1 Operating system and device
management
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/systems
GG24-4484-00 AIX Storage Management (Redbook)
SG24-5422-00 AIX Logical Volume Manager from A to Z: Introduction
and Concepts (Redbook)
SG24-5433-00 AIX Logical Volume Manager from A to Z:
Troubleshooting and Commands (Redbook)

© Copyright IBM Corp. 2009, 2011 Unit 7. LVM metadata and related problems 7-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 8. Disk management procedures

What this unit is about

This unit describes different disk management procedures:
• Managing quorum with mirrored logical volumes
• Disk replacement procedures
• Procedures to solve problems caused by an incorrect disk
replacement

What you should be able to do

After completing this unit, you should be able to:
• Manage volume group quorum issues
• Explain the physical volume states used by the LVM
• Replace a disk under different circumstances
• Recover from a total volume group failure

How you will check your progress

Accountability:
• Lab exercises
• Checkpoint questions

References
Online AIX Version 7.1 Command Reference volumes 1-6
Online AIX Version 7.1 Operating system and device
management
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/systems
GG24-4484 AIX Storage Management (Redbook)
SG24-5432 AIX Logical Volume Manager from A to Z: Introduction
and Concepts (Redbook)
SG24-5433 AIX Logical Volume Manager from A to Z:
Troubleshooting and Commands (Redbook)

© Copyright IBM Corp. 2009, 2011 Unit 8. Disk management procedures 8-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 9. Install and cloning techniques

What this unit is about

This unit describes techniques to reduce the size of a maintenance
window. Specific techniques are taught for installing system updates
while cloning the rootvg.

What you should be able to do

After completing this unit, you should be able to:
• Apply maintenance using the alternate disk copy technique
• Apply maintenance using the multibos technique

How you will check your progress

Accountability:
• Checkpoint questions
• Lab exercise

Reference
Online AIX Version 7.1 Command Reference volumes 1-6
Online AIX Version 7.1 Operating system and device
management
Online AIX Version 7.1 Installation and migration
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/systems
SG24-2014 AIX Version 4.3 Differences Guide (Redbook)
SG24-5765 AIX 5L Differences Guide: V 5.2 Edition (Redbook)
SG24-7463 AIX 5L Differences Guide: V 5.3 Edition (Redbook)
SG24-7414 AIX 5L Differences Guide: V 5.3 Addendum
(Redbook)
SG24-7559 IBM AIX Version 6.1 Differences Guide (Redbook)

© Copyright IBM Corp. 2009, 2011 Unit 9. Install and cloning techniques 9-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 10. Advanced backup techniques

What this unit is about

This unit describes techniques to ensure data integrity and
consistency while executing on-line backups.

What you should be able to do

After completing this unit, you should be able to:
• Explain factors related to on-line backup consistency
• Use JFS split mirror to backup file system data
• Use a snapshot volume groups to backup filesystem data
• Use JFS2 snapshot to back up file system data
• Explain AIX considerations in using SAN copy facilities

How you will check your progress

Accountability:
• Checkpoint questions
• Lab exercise

Reference
Online AIX Version 7.1 Command Reference volumes 1-6
Online AIX Version 7.1 Operating system and device
management
Online AIX Version 7.1 Installation and migration
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/systems
SG24-2014 AIX Version 4.3 Differences Guide (Redbook)
SG24-5765 AIX 5L Differences Guide: V 5.2 Edition (Redbook)
SG24-7463 AIX 5L Differences Guide: V 5.3 Edition (Redbook)
SG24-7414 AIX 5L Differences Guide: V 5.3 Addendum
SG24-7559 IBM AIX Version 6.1 Differences Guide (Redbook)

© Copyright IBM Corp. 2009, 2011 Unit 10. Advanced backup techniques 10-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 11. Diagnostics

What this unit is about

This unit is an overview of diagnostics available in AIX.

What you should be able to do

After completing this appendix, you should be able to:
• Use the diag command to diagnose hardware
• List the different diagnostic program modes

How you will check your progress

Accountability:
• Activity
• Checkpoint questions

References
Online AIX Version 7.1 Understanding the Diagnostic
Subsystem for AIX
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/eserver/

© Copyright IBM Corp. 2009, 2011 Unit 11. Diagnostics 11-1

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
V6.0
Course Quick View with Full TOC

Uempty Unit 12. The AIX system dump facility

What this unit is about

This unit explains how to maintain the AIX system dump facility and
how to obtain a system dump.

What you should be able to do

After completing this unit, you should be able to:
• Explain what is meant by a system dump
• Determine and change the primary and secondary dump devices
• Create a system dump
• Execute the snap command
• Use the kdb command to check a system dump

How you will check your progress

Accountability:
• Checkpoint questions
• Lab exercise

References
Online AIX Version 7.1 Command Reference volumes 1-6
Online AIX Version 7.1 Kernel Extensions and Device Support
Programming Concepts (Chapter 16. Debug Facilities)
Online AIX Version 7.1 Operating system and device management
(section on System Startup)
Note: References listed as “online” above are available at the
following address:
http://publib.boulder.ibm.com/infocenter/systems

© Copyright IBM Corp. 2009, 2011 Unit 12. The AIX system dump facility 12-1
Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.