Test 000-104: AIX 6.

1 Administration
What you see in Red are the objectives of the exam defined by IBM: http://www-03.ibm.com/certify/tests/obj104.shtml I have tried to add notes to each item to some extent. This is not a replacement for IBM documents or courses, but can be used as a wrap-up for the exam or as a reference for some admin tasks. The document was not intended for public use in the first place, that is why you will typo mistakes, formating or other problems in it. Hope these notes help you pass the exam with a better score :) Regards, Mehdi Salehi You can reach me at md_salehi@yahoo.com or ezzobad@gmail.com or http://it.toolbox.com/people/mehdisalehi/ Note: References are mostly IBM redbooks, man pages and other freely-available IBM web resources.

Backup and Recovery (5%) a. Recover from a lost root password 1. Boot the LPAR from AIX media, mksysb tape or NIM server. The boot resource should have the same version and TL as the system you want to recover. For example, an AIX 6.1 with TL6 cannot be recovered by AIX 6.1 TL2 media or NIM resource. 2. Choose Start Maintenance Mode for System Recovery . 3. Select Access a Root Volume Group. A message displays explaining that you will not be able to return to the Installation menus without rebooting if you change the root volume group at this point. 4. Type 0 and press Enter. 5. Type the number of the appropriate volume group from the list and press Enter. 6. Select Access this Volume Group and start a shell by typing 1 and press Enter. 7. At the # (number sign) prompt, type the passwd command at the command line prompt to reset the root password. For example: # passwd Changing password for "root" root's New password: Enter the new password again: 8. To write everything from the buffer to the hard disk and reboot the system, type the following: # sync;sync;sync;reboot

b. Backup AIX OS and data using AIX commands (mksysb, mkcd, tar, backup, etc) mksysb: Backup to tape (Note: Not all tape drives are bootable!): # mksysb -iXV /dev/rmt0 Backup to filesystem (the filesystem path can be local or NFS-mounted): # mksysb -iX /backups/mksysb31Mar2011.mksysb Backup a client from NIM server (Note: /mksysbs in the following command should be NFS exported to testlpar): # nim -o define -t mksysb -a server=master -a location=/mksysbs/testlpar31Mar2011.mksysb -a source=testlpar -a mk_image=yes -a mksysb_flags=XeA testlpar_31Mar2011_mksysb Check the NIM resource in NIM server: # lsnim -t mksysb testlpar_31Mar2011_mksysb resources

mksysb

Note: mksysb only backs up files and directories in rootvg that are mounted. There are other methods to clone an AIX systems: o Alternate Disk Install o Tivoli Sysback o Taking mirror disks of rootvg to another system! o And probably more Mksysb image can be extracted from tape to be used in NIM server.: o First you should find the block size of the tape when the mksysb has been performed: # chdev -l rmt0 -a block_size=512 # tctl -f /dev/rmt0 rewind # restore -s2 -xqvf /dev/rmt0.1 ./tapeblksz # cat tapeblksz 1024 NONE It means the mksysb backup has been made using block size of 1024. # chdev -l rmt0 -a block_size=1024 # tctl -f /dev/rmt0 rewind # dd if=/dev/rmt0.1 of=/mksysbs/mksysb1 bs=1024 fskip=3

It is possible to show information about a mksysb image: # lsmksysb -lf /tmp/mksysbfile <-- this will show infromation about filesystems and OS level of the image. (Actually lsmksysb is a soft link to listvgbackup. It means you could use "listvgbackup -lf /tmp/mksysbfile" instead of above command as well) savevg and restvg: - The volume group should be vary-on and filesystems should be mounted. - This will backup testvg into a file called vgbackup1: o # savevg -if /backups/vgbackup1 testvg

In order to exclude files, edit /etc/excluce.testvg. If you destroy the volume group, it can be restored by restvg: o # restvg -f /backups/vgbackup1 hdisk1

mkszfile and mkvgdata: When you use -i switch with mksysb and savevg, they call mkszfile and mkvgdata respectively. It will create /image.data for rootvg, /tmp/vgdata/testvg/testvg.data for a user-created volume group like testvg and /tmp/wpardata/wpar1/image.data for a workload partition called wpar1. If you need to change the characteristics of the restored volumes group, above files should be edited and then mksysb, savevg are used without -i switch. Note: /usr/bin/mkszfile is a shell script that has two aliases: mkvgdata and mkwpardata. The script runs differently based on the name of invoker file: NAME=`/usr/bin/basename $0` if [ $NAME = "mkszfile" ] then set -- `${getopt} XfmN $*` # mkszfile options

savewpar savewpar cannot be used to create bootable tapes. The command switches are very similar to savevg. Example: # savewpar ief /backups/wpar1backup wpar1 Note: How to exclude files from and volume group or wpar backup: - Create a file called /etc/exclude.rootvg, /etc/exclude.testvg or /etc/exclude.wpar1 - Put the pattern you would like to exclude: ^./home excludes /home filesystem testfs excludes any file or directory that grep finds testfs pattern it their path. - # mksysb eX /mksysbs/newbackup - # savevg ief /backups/vgbackup1 testvg - # savewpar ief /backups/wpar1backup wpar1 Another way to exclude filesytems in a backup is to remove filesystem and its associated logical volumes information from image.data (of rootvg or a workload partition) or testvg.data for a user-created volume group named testvg.

mkcd /mkdvd - Create multi-volume CDs from a mksysb, savevg, or savewpar backup image. - Can generate a new backup or alternatively use existing mksysb, savevg or savewpar image. - Generate CD or DVD images o Images can be burnt now

o Images can be saved for later use # mkdvd d /dev/cd0 bootable rootvg backup # mkdvd d /dev/cd0 W wpar1 # mkdvd S I /backups/ -C /backup -W wpar1 stop to burn and keeps the images in /backups. # mkdvd SI /backups C /bakcups v testvg There are so many command switches. You can use smit for more convenience.

Note: mkdvd is an alias to mkcd tar # tar cvf /dev/rmt0 /data backs up /data tree to rmt0 tape # tar tvf /dev/rmt0 lists the table of content # tar xvf /dev/rmt0 extracts (restores) /data Note: - When you use relative path, be careful when you restore the backup. You should go the same directory to restore it. - Tar can backup to file: o # tar cvf /backups/newbackup.tar /data - You can use tar without the dash charater -: o # tar tvf /dev/rmt1 - You can backup many files and create a very big tar file, but each file cannot be bigger than 8GB. To dodge this problem you can use GNU tar. I have tested it with files of 80GB, and it did not complain.

backup Backup files by name: - Use -i flag. - # find /home/Salehi | backup -ivqf /dev/rmt0 Backup filesysems by i-node: - Need the filesysem to be un-mounted. - backup -2 means level 2. If you use -u, it performs an incremental backup. u means update /etc/dumpdates - # backup -1 -u -f /dev/rmt0 /data c. Restore AIX OS and data using AIX commands, including listing backup media contents (restvg, restore, tar, etc) To restore a mksysb tape, just try boot from it. If the tape is not bootable, boot from AIX DVD and then in SMS menus try to restore the mksysb by selecting the tape drive: Normal Mode Boot Yes Start Maintenance Mode for System Recovery Install from a System Backup restvg # restvg -f /backups/vgbackup1 hdisk1

restore - To show the contents of a backup: o # restore Tvqf /backups/mydata.bak - To extract all mine directory and its contents: o # restore xvqf /backups/mydata.bak /data/mine/ restwpar # restwpar -f /backups/wpar1.bak -n wapr2 -d /newbasedir

System Initialization and Boot (7%) a. Describe and modify the /etc/inittab and rc files b. Describe the different run levels and boot modes a,b,c and h are not true runlevels: they are processed only by telinit (not by init) A process started by these runlevels is not killed when init command changes runlevels. c. Use commands to manage the boot list and create boot logical volumes (incl. changing the boot list) d. Describe the boot process (BIST, POST, mounts, cfgmgr) AIX boot process: 1. POST and hardware checking 2. System ROS locates and loads the bootstrap code. It is operating system independent. 3. Software ROS (bootstrap) creates RAMFS, locates the BLV and turns control to it. 4. RAM filesystem includes a reduced version of ODM (such as PdDv), rc.boot 5. Base devices are configured and init process will be started from RAMFS. 6. There is still no rootvg! But disks have been configured and are ready. Now rc.boot will be called three times: 7. Phase1: a. init process is already running. So it forks rc.boot 1 b. ODM is copied to RAMFS from BLV c. cfgmgr f configures the necessary items to have rootvg disks. 8. Phase 2: a. Rootvg is varied on. b. fsck f /dev/hd4 (root filesystem) c. hd4 is mounted on /mnt in RAMFS d. /usr and /var are checked and mounted e. /var is checked and mounted f. If system has been dumped before, copycore command copies the dump from /dev/hd6 (default) to /var/adm/ras. g. /var is unmounted. h. The primary paging space h6 is activated. i. All /dev files are copied from RAMFS to disk

All customized ODM files from the RAM file system are copied to disk. Both ODM versions from hd4 and hd5 are now synchronized. k. Root filesystems are mounted. 9. Phase 3: a. Rc.boot 3 (from disk) b. /tmp is mounted c. Syncvg rootvg d. Cfgmgr p2 for the rest of devices for normal boot. For service mode p3 is invoked. e. Cfgcon configures the console and boot messages are sent to the console f. ODM of BLV and / are synched. g. Syncd and errdemon are started. h. Init turns the control to the next line of inittab e. Interrupt the boot process and use SMS f. Describe booting from different media (disk, network, tape, cd)

j.

g. Perform system or partition startups, shutdowns and reboots bootlist: Displays and alters the list of boot devices available to the system bootlist has some modes: normal: When the system is booted in normal mode service: When the system is booted in service mode prevboot: Some hardware platforms may attempt to boot from the previous boot device before looking for a boot device in one of the other lists. To show the normal bootlist: # bootlist -m normal -o To set the normal mode bootlist: # bootlist -m normal cd0 hdisk0 To clear (invalidate) the service mode bootlist: # bootlist -m service i When a partition is activated, you can choose the boot mode: Normal: Uses normal mode bootlist stored in NVRAM SMS: Boot process stops at System Management Services menus. DIAG_STORED: Uses service mode bootlist and eventually shows diag menus. DIAG_DEFAULT: Like DIAG_STORED, it is used for diag, but uses default boot list (not what you have set using boot -m service)

OPEN_FIRMWARE: System boots to Open Firmware (used by service personnel) Useful shutdown switches: # shutdown -l (creates /etc/shutdown.log for diagnostics. -l stands for log). # shutdown -Fr (fast reboot) System and Device Configuration (9%) a. Add or remove devices (printers, tape, adapters, using cfgmgr, etc) Add a device: Physically attach the device to the system. (The device may be hot-pluggable or not) If the system is powered-off, power it on. It will run cfgmgr by default. Otherwise, run cfgmgr which will introduce the device into AIX ODM. o If the device driver of the attached device does not exist in the system, install it explicitly or have cfgmgr to install it: # cfgmgr -i /dev/cd0 Remove a device: # rmdev -l rmt0 # rmdev -dl rmt0 (notice! This command only unconfigures the device, and do not removes it) (removes the device from ODM)

# rmdev -Rdl fcs0 (removes fcs0 and all its children recuresively) # rmdev -p fcs0 (just removes the children, not fcs0 itself) b. Determine / chance device attributes, including WWN, MAC addresses, etc. (lsdev, chdev, lscfg, lsattr) Chdev: Changing the attributes of a device if it is busy: # chdev -l ent0 -a ... -P (P stands for permanent) Determine WWPN or FC adapter: # fcstat fcs1 | grep -i "world wide port name" World Wide Port Name: 0x10000000C97A34BF Or: # # lscfg -vl fcs1 | grep -i "network address" Network Address.............10000000C97A34BF Determining WWNN of FC adapter: # fcstat fcs1 | grep -i "world wide node name" World Wide Node Name: 0x20000000C97A34BF Or:

# lscfg -vl fcs1 | grep -i z8 Device Specific.(Z8)........20000000C97A34BF Determining Ethernet adapter MAC address: # entstat -d ent0 | grep -i "hardware address" Hardware Address: 00:14:5e:53:9d:40 Or: # lscfg -vl ent0 | grep -i "network address" Network Address.............00145E539D40 c. List, define and change paging space List paging space: # lsps -a shows detailed output # lsps -s shows a summary # mkps -s 1 -n -a testvg hdisk1 defines a paging space with one PP, starts now and at restart # chps -s 1 paging00 adds one PP to the paging space # chps -d 1 paging00 removes one PP from the paging space # swapon /dev/paging00 activate the paging space now # swapoff /dev/paging00 # rmps paging00 remove the paging space d. Configure and manage print subsystem (print queues, default printer, print job management) e. Configure system environment (timezone, /etc/environment, etc.) f. Add / remove disks (including data migration tasks, using cfgmgr)

Network Administration (9%) a. Configure the network (TCP/IP daemons, /etc/hosts, hostname, ifconfig, route, /etc/resolv.conf, etc/netsvc.conf, /etc/ntpd.conf) /etc/hosts: You can add, change or delete entries from this file by hostent command. (Manual editing is still available). This adds a record to /etc/hosts with primary hostname of salehi and an alias named mypc: # hostent a 10.0.62.14 salehi mypc To show the record associated with Salehi: # hostent s salehi 10.0.62.14 salehi mypc Reserved host names: timeserver If you set timeserver in /etc/hosts, you get run setclock to get its time and set it to the current system.

printserver Identifies the default host to receive print requests. hostname: - hostname command can show or temporarily set the hostname of a system: o # hostname newhostname (next reboot will roll it back. It is not permanent.) - Another way to permanently set hostname: o # chdev -l inet0 -a hostname=newhostname o This will not change /etc/hosts - Another way: o # smit mkhostname o This will not change /etc/hosts - Another way: o # mktcpip -h newhostname -a 10.0.84.79 -m 255.255.255.0 -i en0 o This will change /etc/hosts. (Actually adds the new host name as an alias of previous value in /etc/hosts.) Conclusion: When you change hostname, always check /etc/hosts.

ifconfig: To list all interfaces that are up with details: # ifconfig -au To add IP to en0: # ifconfig en0 10.1.2.3 netmask 255.255.255.0 up To bring a network interface down: # ifconfig en0 down Note: Changes made by ifconfig will be gone in next restart. route: To list the routing table: # netstat nr To find the default gateway: # netstat -nr | grep default | awk '{print $2}' To establish a default gateway: # route add 0 192.168.1.1 Add route to a destination (like 11.25.12.1) via a gateway (like 10.10.10.1): # route add 11.25.12.1 10.10.10.1 To reach a network (like 50.1.3.0) via a gateway like 172.16.16.1 via en0:

# route add -net 192.168.10.0 10.0.62.14 interface 0 Or: # chdev -l inet0 -a route=net,-hopcount,0,,-if,en0,,,,-static,50.1.3.0,172.16.16.1 To delete above route: # route delete -net 50.1.3.0 # chdev -l inet0 -a delroute=net,-hopcount,0,,,50.1.3.0,172.16.16.1 Note: The effect of route command is not permanent. Sometimes it is desirable to set routing via a script when needed (like in HACMP environment). If you need to make it permanent, use chdev -l inet0 instead. resolv.conf: AIX uses some methods to map host names to their IP address: - /etc/hosts - DNS - NIS - LDAP If /etc/resolv.conf does not exist: it means the network is flat and therefore /etc/hosts will be used for name resolution. If /etc/resolv.conf exists: We have domain network and therefore resolver algorithm will be used. File format: A domain entry tells the resolver routines which default domain name to append to names that do not end with a . (period). There can be only one domain entry. This entry is of the form: domain my.domain.com search is another entry of this file that is mutually exclusive with domain. With search you can specify many domains to search within when you are resolving a name. The first domain in the search list, is default domain. nameserver entry specifies the remote domain name server. - The address is dotted decimal - You can specify more than one name server: nameserver 192.9.21.1 nameserver 192.9.21.2 Note: - If both domain and search entries exist, the one that appear last will be considered. - If there is no default domain in /etc/resolv.conf, you should set it in the hostname. - If you use LDAP, /etc/resolv.ldap should be configured. - Name resolution order is specified in irs.conf and netsvc.conf and NSORDER environment variable. NSORDER overrides the settings of netsvc.conf and netsvc.conf overrides irs.conf. netsvc.conf: It is used to specify the ordering of name resolution. Syntax:

hosts = value [, value] alias = value [, value] Sample: #checks /etc/hosts and then DNS for name resolution: Hosts = local, bind # checks /etc/aliases and then NIS to resolve aliases for sendmail: alias = files, nis

/etc/aliases: /etc/aliases is a link to /etc/mail/aliases

Contains the required aliases for the sendmail command.

moi:

salehi

NSORDER: If NSORDER environment variable is set, it overrides the settings of netsvc.conf and irs.conf Example: # export NSORDER=bind,nis,local ntp.conf: # startsrc -s xntpd # lssrc -ls xntpd | grep peer Sys peer: no peer, system is insane In ntp.conf: - Add this: server 127.127.1.0 - and comment this: #broadcastclient # stopsrc -s xntpd # startsrc -s xntpd -a x

insane means ntp configuration is wrong!

(-x can be very important)

Wait for one or two miutes and then: # lssrc -ls xntpd | grep peer Sys peer: 127.127.1.0 flags: (configured)(refclock)(sys peer) On ntp client side: # ntpdate d node1 If offset is more than 1000 seconds, change the time date manually and then try above command again. Note: You can set the client to automatically sync the time with your server. - Add a server entry in /etc/ntp.conf, but this time the address of your timeserver.

Uncomment broadcastclient # stopsrc -s xntpd # startsrc -s xntpd -a x (-x can be very important)

In order to start xntpd in system startup, change /etc/rc.tcpip. This can be done both in client and server. b. Configure network security (/etc/hosts.equiv, .rhosts, etc.) First /etc/hosts.equiv and then $HOME/.rhosts will be checked to see whether the remote r-command request is from a trusted host or not. Sample: toaster # all users from toaster are allowed machine1 bob # only bob from machine1 + lester # user lester from all machines tron joel # user joel from tron host is not allowed. tron # all userd from trom are allowd. Note: - For root user, only /.rhosts is checked. - If /etc/hosts.equiv and $HOME/.rhosts have write permission for group or others, password will be asked! - The deny, or - (minus sign), statements must precede the accept, or + (plus sign), - statements in the lists - Generally it is not secure to use this kind of password-less communication. You can use SSH key pairs, instead. c. Verify network availability and debug network problems (ping, ifconfig, netstat, tcpdump, iptrace) tcpdump: It prints the headers of packets on a network interface. Example: # tcpdump -i en0 To print all packets arriving at or departing from Salehi: # tcpdump host salehi Iptrace: It provides interface-level packet tracing for IP protocol. It generates a log file that can be very big. iptrace can be started by issuing iptrace command itself or by SRC. If not started by SRC, the process should be stopped by kill -15. (-15 is SIGTERM or software termination signal). Example: # startsrc -s iptrace -a "/tmp/nettrace" # stopsrc -s iptrace # iptrace -i en0 -p telnet -s airmail /tmp/telnet.trace

# kill -15 234343 d. Understand and configure Etherchannel and teaming e. Configure NFS (/etc/exports/, biod, nfsd, showmount, etc.) /etc/exports: If this file is present, at system startup /etc/rc.nfs brings up nfsd and mountd. The entries of this file are like this: Directory options Example: /soft # exports to the world /usr2 -access=hermes:zip:tutorial # exports only to these systems /usr/tps -root=hermes:zip # root access only to these systems Important daemons and commands: - nfsd: o Services client requests for file system operations. o Each daemon handles one request at a time. You can tune the max threads by chnfs or chssys. mountd: o It is an RPC that answers a client request to mount a filesystem. chnfs: o # chnfs -n 10 -I (sets the number of nfsd daemons). exportfs: o Exports and unexports directories to NFS clients. o # exportfs -a (exports all in the /etc/exports) o # exportfs /dir1 (exports only /dir1 which is in the /etc/exports) o # exportfs -i /dir2 (exports only /dir1 which is not in the /etc/exports) o # exportfs u /dir2 (unexports /dir2)

Note: You cannot export either a parent directory or a subdirectory of an exported directory within the same file system. biod: It handles client requests for files. It is an old daemon and might be removed in future AIX releases. showmont: # showmount -a (shows all clients that have mounted something on this server) # showmount -e nfssrv1 (show which filesystems are exported from nfssrv1) /etc/xtab: Contains entries for currently mounted NFS directories. exportfs -u removes entries from this

file. f. Configure and use CIFS (very basic) Install bos.cifs_fs package in AIX and then smit cifs_fs. Thats it! This will enable AIX to mount Windows shared directories. These ports should be opened: 137,138,139 and 445 Security and User Management (7%) a. Add, delete, change user and group accounts # mkuser -a mehdi <== mehdi will be admin # mkuser -R LDAP Nava <== Nava will be authenticated by LDAP # chuser shell=/usr/bin/bash mehdi <== changes the user's shell How to reset the failed login count: # chsec -f /etc/security/lastlog -a "unsuccessful_login_count=0" -s mehdi b. Describe and modify user and group management related files, profiles, and set or change the shell environment (/etc/security/user, /etc/security/limits, /etc/security/passwd, /etc/profile/, .profile) c. Demonstrate in-depth knowledge of the login process (is getty running, order of the environment being set, etc.) Login process: 1- When getty which is a long running process - detects a connection, it prompts for a username and runs the login program to authenticate the user. So, getty is the first step started from inittab: cons:0123456789:respawn:/usr/sbin/getty /dev/console 2- getty prints a herald message from /etc/security/login.cfg to get the user name from input. 3- getty calls login process to check whether password is needed to login or not. If password is needed, another prompt will ask for it. Note: If the second field of /etc/passwd is null, the user can login without password: testuser::208:1::/home/testuser:/usr/bin/ksh This method works only with telnet. ssh asks always for password. 4- Login process do the validation process a. If login fails, a record is added to /etc/security/failedlogin b. If login is successful:
a. b. c. a. /etc/environment /etc/security/environ /etc/security/limits /etc/security/user

b. /etc/profile

c. $HOME/.profile (or .dtprofile for CDE) b. Set permissions (in more depth than operator) c. Configure RBAC (role-based access control) The majority of the Enhanced RBAC commands are included in the bos.rte.security fileset. Authorizations are assigned to roles, which may then be assigned to user. KST stands for Kernel Security Tables o lskst Enhanced RBAC security database to be stored in LDAP o System-defined authorizations cannot be stored in LDAP and will remain local to each client system. If enhanced_RBAC of sys0 is true, RBAC is enhanced. You can change it to false to go back to Legacy RBAC. Predefined roles: o ISSO (Information System Security Officer) The most powerful role o SA: (System Administrator) Cannot change passwords o SO: (System Operator) To list the roles: - # lsrole ALL | awk '{print $1}' AccountAdmin BackupRestore DomainAdmin FSAdmin SecPolicy SysBoot SysConfig isso sa so Add role to a user: (for example add shutdown and reboot privilege to user salehi) - # lssecattr -c /usr/sbin/reboot | awk '{print $2}' accessauths=aix.system.boot.reboot # lssecattr -c /usr/sbin/shutdown | awk '{print $2}' accessauths=aix.system.boot.shutdown There might be an existing role that contains above authorizations: # lsrole ALL | grep aix.system.boot.reboot | awk '{print $1}' SysBoot Assign the role: # lsuser -a roles salehi salehi roles=SysBoot

# chuser roles=SysBoot Salehi # lsuser -a roles salehi salehi roles=SysBoot The user itself can list the roles: # su - salehi -c "rolelist" SysBoot System Boot Administration Activate the role: - If the user does not activate a role, it is still an ordinary user without any role. - # swrole SysBoot (switches to SysBoot role) - # swrole ALL (switches to all user roles) - # rolelist e (lists effective roles) SysBoot System Boot Administration Role authentication: Be default user should provide password to activate a role. Because auth_mode=INVOKER. # lsrole -a auth_mode SysBoot SysBoot auth_mode=INVOKER You can change it: # chrole auth_mode=NONE SysBoot # lsrole -a auth_mode SysBoot SysBoot auth_mode=INVOKER Create a user-defined role: The goal is to assign a role to a user to enable him to change cron settings: # lsauth ALL | grep cron | cut -f1 -d' ' aix.system.config.cron Only sa (system administrator) has this authorization: # lsrole ALL | grep aix.system.config.cron | cut -f1 -d' ' sa So we need to define a role: # mkrole authorizations="aix.system.config.cron" cronRole Assign the role to the user: # chuser roles=cronRole salehi Read the RBAC security database files and load the information from the database files into the Kernel Security Tables (KST): # setkst Now Salehi can change roots crontab: # su Salehi # swrole ALL

# crontan e root Another example: Grant write access to /etc/hosts to operator2 (you need to create a new authorization for it): root:/> mkauth newauth root:/> setsecattr -f writeauths=newauth /etc/hosts root:/> mkrole authorizations=newauth etchostsRole root:/> chuser roles=etchostsRole operator2 root:/> setkst root:/> su - operator2 operator1:/home/operator2> swrole ALL operator1:/home/operator2> vi /etc/hosts

Install and Maintain AIX (11%) a. Determine correct installation source (CD/DVD, NIM, cloning, alternate disk install, etc) Minimum memory supported by AIX 6.1 is 265 MB. b. Determine correct installation type (preservation, migration, new/complete overwrite) New and complete overwrite destroys everything on the specified disks. Migration changes the AIX version and/or release (like from 5.3 to 6.1) Preservation method keeps user data in rootvg intact. But removes /usr, /, /var and /tmp c. Install, check and remove updates, TLs and fixes. Describe lpp statuses and tasks (commit, apply, or reject using lslpp), and debug install errors using lppchk # installp -r <package_name> <== rejects an applied software # installp -c all <== commits all # installp -C <== cleanup after a fialed or interrupted software install # installp -acgYd /dev/cd0 cluster.* (install, commit, requisite install, accept license, path of source media)

d. Describe various options to acquire updates and fixes (SUMA, FLRT) List the SUMA global configuration settings: # suma c Change SUMA global configuration settings: # suma -c -a HTTP_PROXY=http://user:pass@proxysrv:8080 Download critical fixes now: # suma -x -a Action='download' -a RqType=' Critical' To see the difference between available fixes and what you in /soft/AIX/6.1/AIX61TL6: # suma -x -a Action='Preview' -a DLTarget='/TL' -a FilterDir='/soft/AIX/6.1/AIX61TL6' FLRT stands for Fix Level Recommendation Tool an IBM useful page.

e. Install additional IBM and Open Source licensed program products (rpm, rte, bff, etc.) f. Install and configure a basic NIM environment (what it is and what must be configured) nimconfig: (configures the nim master. requires bos.sysmgt.nim.master) To define a NIM master only: # nimconfig -a netname=NIMnet0 -a pif_name=en0 niminit: (configures the nim client) # niminit -a name=testlpar -a master=nimsrv1 -a pif_name=en0 -a netboot_kernel=mp nim: (performs operations on NIM resources) # nim -o allocate -a spot=spot1 -a lpp_source=lppAIX61 nimclient1 # nim -Fo reset nimclient1 # nim -Fo deallocate -a subclass=all testlpar Lots of operations are possible, like: define, change, create, restvg, ... nimclient: (performs NIM operations in NIM client side) # nimclient -l (shows the resrouces) # nimclient -Fo reset (resets the NIM client) g. Obtain and validate system and device firmware, including considerations for 'deferred' and 'concurrent' maintenance. Concurrent update: Firmware that can be applied and activated on running systems. Deferred update: Firmware can be concurrently applied but contains some fixes that can't be activated until the next IPL because the fixes affect the IPL path. Disruptive upgrade/update: A platform IPL is required to activate. None of the content contained in the release/service pack will be activated until the next IPL. Activated Level of firmware: The level running in memory. Normally when you apply the firmware, it is saved in NRAM, but in next IPL it will be loaded to memory. Accepted Leve of firmware: The level saved on p-side of flash. Logical Volume, File and Filesystem management (7%) a. Enlarge and reduce file systems b. Describe and differentiate between physical volumes and LVMs, logical volumes, physical and logical partitions, and physical disk and physical partition size.

c. Manage Volume Groups including mirroring (mkvg, varyonvg, varyoffvg, extendvg, exportvg, importvg, lsvg) Volumg group quorum: # chvg Qn testvg <== truns off quorum If quorum if set to "y", when the volume group loses quorum of VGDAs, it will be automatically varied off. If a volume group loses its quorm of disks, it can be varied on only force (varyonvg -f) d. Describe and manage different types of Logical Volumes, including mirroring. e. Describe and manage different types of filesystems and different logging methods (mkfs, chfs, fsck, mount, snapshot, etc.) # umount -f <== forces the umount, even if the path busy or for remote filesysems if the remote server is not present. # fcsk -p <== Does not display messages about minor problems but fixes them automatically. mounting an ISO image: Method1 (for older AIX versions): Create a logical volume, dd the ISO image to the LV, then mount the LV: # mklv -y dvd_lv testvg 5G # dd if=isofile of=/dev/dvd_lv bs=1m # mount -v cdrfs -o ro /dev/dvd_lv /mnt How to unmount: umount command is used to unmount the image. Method2 (recommended): Using loopback device in AIX 6.1 TL4+ and VIOS: # mkdev -c loopback -s node -t loopback # this creates loop0 once forever. # lsdev -Cc loopback loop0 Available Loopback Device # loopmount -i /soft/TSM/TSMserver.iso -l loop0 -o "-V cdrfs -o ro" -m /mnt How to unmount: If you unmount the image using umount command, loop0 device will not be unconfigured. You can use loopumopunt instead: # loopumount -l loop -m /mnt mounting an USB flash: snapshot: Split-mirror backup: # chfs -a snapdir=/backup -a copy=3 /testfs Now you can backup /backup. When you remove /backup, /testfs will be resynced automatically

which might take a very long time with unwanted I/O load. Question: Is there any limitation for the number of snapshots of a filesystem? something like 15 or 16? Yes: The maximum number of external snapshots per file system is 15, while the maximum number of internal snapshots per file system is 64. There is another method which uses "snapshot" command and used copy-on-wirte algorithm: Changes will go to the snapshot storage. From AIX 6.1 onwards, you can use internal snapshots, it means the space to store snapshot is inside the filesystem itself. Create external snapshot: # mklv -y newsnaplv -t jfs2 datavg 4 # snapshot -o snapfrom=/mksysbs newsnaplv <== newsnaplv is the snapshot device or # snapshot -o snapfrom=/mksysbs -o size=128MB <== create the snapshot LV automatically Verify: # snapshot -q /mksysbs Snapshots for /mksysbs Current Location /dev/newsnaplv * /dev/fslv06 * means current snapshot

512-blocks 2097152 524288

Free Time 2096384 Mon May 16 12:37:13 2011 523520 Mon May 16 12:38:37 2011

<==

you can mount a snapshot: # mount -o snapshot /dev/fslv06 /mnt /mnt will contain the contents of /mksysbs when you created the snapshot. (remember the copy-on-write method). It is mounted as read-only by default. How to rollback: <== this will remove the snapshot You have changed something in /mksysbs filesystem and want to rollback: # umount /mksysbs # rollback -v /mksysbs /dev/fslv06 Delete the snapshot: # snapshot -d /dev/fslv06 Note: Internal snapshot should be enabled only at filesystem creation time: # crfs -v jfs -m /testfs -g rootvg -A yes -a isnapshot=yes -a size=1G copcy some file to /testfs. # snapshot -o snapfrom=/testfs -n monsnap # rollback -v -n monsnap /testfs Shrinking filesystem and defragfs with a snapshot is not supported.

In order to backup the snapshot of a filesystem, use "backsnap" command. f. Configure and manage symbolic and hard links Hard link: Two file names that refer to the same i-node - Source and target should be in the same filesystem - ln: cannot hard link directory (only files) - # ln source target - If you remove source or target, the other one still refers to the i-node and works fine. I-node will be removed if all references (links) are deleted. Soft/symbolic link: - points to the name of source file/directory, not the i-node - can be used across filesystems - # ln -s source target - If source is removed, target will become a dangling reference (= a pointer that points to something that does not exist). g. Demonstrate understanding of multipath I/O Multipath I/O or MPIO means establishing more than one path between the two ends of an I/O stream like between AIX and a disk subsystem. The purpose of MPIO is to provide more resilience and/or better I/O throughput. - AIX native MPIO supports only failover (and no load balancing) for all MPIO-capable disk subsystems. - Each disk vendor should provide a special device driver to provide more advanced algorithms like round-robin, extended round-robin. Examples are IBM SDDPCM (Subsystem Device Driver Path Control Module), Hitachi HDLM (Dynamic Link Manager), EMC PowerPath and so forth. - AIX native MPIO commands: # lspath # mkpath # chpath # rmpath

Problem Determination and Resolution (15%) a. Use logs to identify problems (errlog, alog, syslog, etc.) b. Use the diag utility c. Use traces, truss, snap and kdb trace: The trace daemon records selected system events. Trace has different data collection modes: - Alternate (default): o All trace events are captured in the trace log file.

o If the log file reaches the max size, file is overwritten from beginning. Circular: o Circular logging occurs within trace buffer. Log file is generated only when trace is stopped. o Useful when user knows when the problem occurs. So, if they stop the trace exactly after they encounter the problem, buffer contains useful information that will be save in log file. o # trace -l Single buffer: o Trace stops when the in-memory trace buffer fills up. o The contents of the buffer are captured in the trace log file. o # trace -f Buffer Allocation: o By default, buffers are allocated from the kernel heap. o If requested size is not fit into kernel heap, it will be allocated in separate segments from pinned memory. o # trace -b or -B

The default trace log file is /var/adm/ras/trcfile. This is a binary file that should be viewed by trcrpt. Running trace in interactive mode: # trace > ! anycommand > q Running trace in background: # trace -a -o /tmp/my_trace_log; anycmd; trcstop trcrpt: Formats a report from the trace log with the format the is implied from /etc/trcfmt. # trcrpt -o /tmp/newfile

truss: truss command is useful for tracing system calls in one or more processes: A simple example: # truss -ea hostname execve("/usr/bin/hostname", 0x2FF22C90, 0x20012ED8) argc: 1 argv: hostname envp: AUTHSTATE=compat TERM=xterm SHELL=/usr/bin/bash SSH_CLIENT=10.0.62.14 1781 22 SSH_TTY=/dev/pts/0 LOCPATH=/usr/lib/nls/loc USER=root ODMDIR=/etc/objrepos MAIL=/usr/spool/mail/root PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java5/jre/bin:/usr/java5/bin: LOGIN=root PWD=/home/salehi LANG=C TZ=CST6CDT PS1=\[\]\u\[\]@\[\]\h\[\]:$PWD\[\]>

SHLVL=1 HOME=/ LC__FASTMSG=true MAILMSG=[YOU HAVE NEW MAIL] LOGNAME=root SSH_CONNECTION=10.0.62.14 1781 10.0.84.79 22 DISPLAY=salehi:0 _=/usr/bin/truss OLDPWD=/ AIXTHREAD_SCOPE=S NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat gethostname(0x2FF22AE4, 256) =0 kioctl(1, 22528, 0x00000000, 0x00000000) =0 testlpar kwrite(1, " t e s t l p a r\n", 9) =9 kfcntl(1, F_GETFL, 0x2FF22FFC) = 67110914 kfcntl(2, F_GETFL, 0x2FF22FFC) = 67110914 _exit(0) As you see, -e could be useful to find out what environment vairiables are passed to a command or program. snap: snap command gathers extensive system configuration information. To gather HACMP information: # snap -e To gather all system configuration except HACMP and create a compressed pax output: # snap ca The output pax file will be stored in /tmp/ibmsupt. snap can be used to restore from dump device: ??? kdb: kdb is an interactive utility that allows for the examining of a system or live dump or a running kernel.

d. Describe and use ODM e. Configure and use system dump devices sysdumpdev -l and os forth... f. Recover from a full file system

g. Troubleshoot common boot LED codes and access a system that will not boot LEDs: 0c0...0c9 and 0cc are all related to dump
LED 201 223-229 551-555-557 552-554-556 553 C40 C41 C42 C43 C44 C45 C46 C47 C48 C49 C50 C51 C52 C53 C54 C55 C56 C57 C58 C59 C61 C62 C63 C64 Description Invalid boot image Invalid boot list Corrupted filesystem or JFS log Corrupted superblock or ODM Invalid /etc/inittab configuration files are being restored Could not determine the boot device Extracting data files from diskette Cannot access the install tape Initializing configuration database for target disks Cannot configure the console Normal installation processing Could not create PVID on disk Prompting for user input Could not create or form the JFS log Creating root volume group No paging devices were found Changing from RAM environment to disk environment /tmp is small for preservation installation Installation BOS or other packages Could not remove an LV in preservation installation Running user-defined customization Failure to restore BOS Displaying message to turn the key Could not copy info from RAM to disk Failure to create boot image Loading debug files Loading data files Failed to load data files

h. Troubleshoot installation hangs and failures i. j. Debug shell script common interpreter problems (ksh, etc) Recover a logical volume

k. Find and correct corrupted filesystems, superblocks, etc.

Process and Performance Management and Tuning (9%) a. Use the system resource manager b. Understand and use Workload Manager (WLM) at a basic level # wlmassign --> Manually assigns processes to a Workload Management class # mkclass -> Creates a Workload Management class # lsclass # chclass # rmclass # lswlmconf # wlmstat # wlmcntrl -->Starts or stops the Workload Manager. # confsetcntrl c. Use cron and at at a detailed level The format of crontab file: minute hour day_of_month month weekday command d. Use tuning tools and parameters (ioo, vmo, no, /etc/tunables, etc) e. Use performance monitoring tools (topas, netstat, vmstat, lvmstat, iostat, svmon, nmon) f. Monitor and change process execution (ps, nice, kill)

Planning and Documentation (11%) a. Understand Workload Partitions (WPARs) and when to use them WPAR products consists of two parts: The part that is included in AIX 6.1 WPAR products consists of two parts: The part that is included in AIX 6.1 Workload Partition Manager. WPAR managre help "Live Application Mobilty" (even automatic mobility) Each WPAR uses /usr and /opt as read-only. WPAR types: System partitoin It is a miniture copy of AIX. Create --> (defined state) --> run (active state) --> stop --> (defined state) --> remove application partition The idea is that we put a WPAR around an application. When the applications start, WPAR is created, and when it stoped, WPAR would be removed.

Basic commands: # mkwpar -n wpar1 # lswpar # startwpar wpar1 # stopwpar wpar1 Applicatioin mobity: chkpwpar <-- checkpoints (or freezed the partitoin to a statefile) restartwpar <-- resumes a WPAR probably on a different machine. When you create a WPAR, in order to mark it as a mobile workload partition you need to specify an NFS server. This NFS server will hold the state of WPAR during mobility. You cannot move a WPAR to a different hardware version (like POWER5 to POWER6). b. Plan HMC configuration (networking, redundancy, users, security, etc.) c. Describe the use and function of VIO d. Partition planning (micropartitioning, memory planning, HEA/IVE, processor allocation, etc) e. Document a system (sysplan, etc) f. Find appropriate resources (info center, key center, etc.)

g. Determine system redundancy requirements (avoiding single points of failure) h. Describe applicability and use of Capacity on Demand Permanent: It is a purchage agreement You cannot turn it off One processor or one GB or memroy Trial CoD 30 contiguous days On/Off CoD Temporary additonal processor or memory Activity is reported monthly to IBM Charged vased on number of days, even one minutes! Monthly charge Utility CoD Similar to on/off, but charge is based on minutes rather than days. For Power6+ Capacity Backup: Reserve capacity for backup server

Works up to 90 days

HMC and Partition Management (6%) a. Apply HMC and Server fixes b. Define, add, remove resources from an LPAR (DLPAR and partition profiles, etc.) c. Backup and restore the HMC d. Use the HMC and ASMI interface, e. Understand and use IVM (options, functions, etc.) f. Configure and use electronic service agent

ESA is a free software on AIX 5.3 TL6+ and if configured properly, sends error information to IBM to aid in problem resolution. ESA client is freely available on all IBM systems plus DS8000. # smit esa_main Starting electronic service agent: # startsrc -s IBM.ESAGENT

Miscellaneous:
multibos: Manipulates multiple versions of BOS in rootvg. It means you have more than one operating system in the rootvg disks. Except /, /usr, /var and /opt, all other filesystems and logical volumes would be shared between BOS instances. It is like alternate disk install, but does not require additional disks. choosing between BOS instances is possible when you set boot list Setup: # multibos -R <== Removes all standby BOS objects # multibos -sXp <==To perform a standby BOS setup operation preview # multibos -sX <==To perform a standby BOS setup operation

# multibos -sXp -M /soft/mksysb1 <==To perform a standby BOS setup operation preview from an existing mksysb # bootlist -m normal -o hdisk0 blv=bos_hd5 pathid=1 hdisk0 blv=hd5 pathid=1 To make sure you are booting from the right instance, compare the boot device when AIX is

starting in SMS with what bootlist shows: # bootlist -m normal -ov 'ibm,max-boot-devices' = 0x5 NVRAM variable: (boot-device=/vdevice/v-scsi@30000002/disk@8100000000000000:4 /vdevice/v-scsi@30000002/disk@8100000000000000:2) Path name: (/vdevice/v-scsi@30000002/disk@8100000000000000:4) match_specific_info: ut=disk/vscsi/vdisk hdisk0 blv=bos_hd5 pathid=1 Path name: (/vdevice/v-scsi@30000002/disk@8100000000000000:2) match_specific_info: ut=disk/vscsi/vdisk hdisk0 blv=hd5 pathid=1 # alog -of /etc/multibos/logs/op.alog <== to view the log # lsvg rootvg -l | grep bos_ bos_hd5 boot bos_hd4 jfs2 bos_hd2 jfs2 bos_hd9var jfs2 bos_hd10opt jfs2

1 10 70 12 13

1 10 70 12 13

1 1 1 1 1

closed/syncd closed/syncd closed/syncd closed/syncd closed/syncd

N/A /bos_inst /bos_inst/usr /bos_inst/var /bos_inst/opt

# multibos -S <== initiates an interactive session to the standby BOS # multibos -Xac -l /TL <== applies a TL on standby BOS How to change back the bootlist: # bootlist -m normal -o hdisk0 blv=hd5

Encrypted filesystem: EFS helps to protect data on filesystem by assigning each user a unique encryption key. When a user requests access to a file, kernel checks the credentials. The cryptographic information is kept in the extended attribute of the file. This is an additional granularity and flexibility to traditional access permissions. - How to enable EFS: # efsenable -av This will create /var/efs directory (that keeps keystores) and alters /etc/security/user and group. - Create two EFS-enabled filesystem: # crfs -v jfs2 -g rootvg -m /sales -a size=100M -a efs=yes # crfs -v jfs2 -g rootvg -m /finance -a size=100M -a efs=yes Make users to access each filesystem: # mkuser saleman; passwd salesman # mkuser financeman; passwd financeman passwd in previous step, causes to create a separate directory (here called keystore) for the user in /etc/efs/users:

# ls /var/efs/users/ total 0 -rw------1 root drwx-----2 root drwx-----2 root drwx-----2 root -

system system system system

0 Apr 26 05:52 .lock 256 Apr 26 06:08 finance 256 Apr 26 05:52 root 256 Apr 26 06:08 sales

demostration: # mount /finance # su -finance # mkdir -p /finance/yearlyreport # chmod -R 777 /finance/yearlyreport look at full permission # efsmgr -E /finance/yearlyreport enables efs for the directory # efsmgr -L /finance/yearlyreport list EFS inheritance is set with algorithm: AES_128_CBC Login back: # su - finance # touch /finance/yearlyreport/anewfile touch: /finance/yearlyreport/anewfile cannot create But you can load the keystore and run a command: # efskeymgr o <thecommand> # efskeymgr o bash this will open a bash session Now you can touch the file. # ls U for security information 2 finance staff drwxrwxrwxe

256 Apr 26 08:29 yearlyreport

Some HMC tips: HMC web acces port is 443 Each POWER system has three users by default in ASM: admin, general and HMC. The HMC user is the one hardware management console uses to be authenticated against when it discovers the machine. Trusted Execution: Trusted Execution is a security feature of AIX 6.1. To some extent it is similar to TCB, but: TCB should enabled at installation phase. TCB checks the integrity in time intervals using cron. TE check the integrity of command when they are invoked.

SEA on HEA: Is SEA possible on HEA in promiscuous mode? Answer: Yes sugroup:

http://www.ibm.com/developerworks/aix/library/au-sugroup/index.html

/etc/objrepos/errnotify: http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.baseadmn/doc/b aseadmndita/HT_baseadmn_missingpv.htm and http://www.blacksheepnetworks.com/security/resources/aix-error-notification.html

Disabling JFS2 logging: # mount -o log=NULL /testfs

Add more . Hope this helps, Mehdi