Chapter 4. Disk 70 performance monitoring tools The following topics are discussed in this chapter: * Logical Volume Manager (LVM) performance analysis, using the 1s1v command, + Journaled File system (JFS) performance analysis tools, using the £itenan and fileplace commands. All topics and tools discussed in this chapter will provide you with a set of methods in determining logical volume manager, file system, and disk /O related performance issues. 4.1 Overview With the AIX operating system, the handling of disk related 1/0 is based upon different functional levels, as shown in Figure 13. Application Level JFs Raw Logical Level Logical Volume Manager Physical Level Disk Drive Adapter Figure 13. Disk, LYM and fe system levels The lowest level is the physical level, that consists of device drivers accessing the physical disks and using the corresponding adapters. The next level is the logical level, managed by the Logical Volume Manager (LVM), Which controls the physical disk resources. The LVM provides a logical mapping of disk resources to the application level. The application level can consist of either the Journaled File System (JFS) or raw access, for example, used by relational database systems. ‘The performance analysis tools discussed in the following section focus on the logical level (LVM) and on the application level (JFS). The monitoring of the physical level is primarily done by using the iostat command, which is described in Section 4.2, "The iostat command’ on page 108. (© Copyright 18M Corp. 2000 107Covering the entire subject of the AIX Logical Volume Manager is beyond the scope of this publication. For more background information on the AIX Logical Volume Manager, refer to AIX Logical Volume Manager, from A to Z: Introduction and Concepts, SG24-5432, as well as the AIX Version 4.3 System Management Guide: Operating System and Devices, SC23-2525, 4.2 The iostat command The Lostat command is a useful tool that provides a first indication of VO related performance problems. The iostat command is capable of reporting CPU statistics, terminal I/O statistics, and disk I/O statistics, that help identity the /O load on individual components, such as hard disks. The information from osvat reports can be used to modify system configurations to improve I/O load distribution between physical disks. The iostat command extracts data from AIX kernel I/O counters in the kernel address space, which are updated at every clock tick (1/100 second) for TTY as well as CPU and I/O subsystem activity. The syntax of the iostar command is as follows: icotat ot nysicalvelime ...] [interval | come] The commonly used flags are provided in Table 8. Tablo 8 Commonly used fags ofthe jostat command Description This flag displays only the disk utlization report. The -< flag is, exclusive of the -t flag. This flag displays only the TTY and CPU usage. The -tflag is exclusive of the -d fag, By using the physicalVolume parameter (by specifying the physical volume (PV) name of the individual disk or CD-ROM), soseat generates an /O report only for the specified PVs. If no PVs are specified, the sostat command generates a report for all drives. The interval parameter specifies the amount of time in seconds between each report. The count parameter specifies the number of I/O reports generated. If the interval parameter is specified without the count parameter, the sostat command generates reports continuously. 108 15M Certficaton Study Guide AIK Pertormance and System TuningAn example of the sostat command is as follows: tty: tin cout avg-cpu: tuser faye F idle ¥ iowait ane el. ot 38.9 0.0 Semact ps tes Kh read Kort, 9.0 0.3 0.0 258032224266, dike on aa 0.0 2sa0ge 1658678, ndiaco 9.0 0.9 8 aeis2 725871 disk on Ls 0.0 974788 1660027 2.0 0.2 0.0 32308040480 tty: tin cout avg-cpur ¥user faye Fidle ¥ jowait 603.5 ss as 0.0 0.0 Simact Kbps read Kb_wetn 6.0 8.0 2e16 o ndiake2 26.0 2.8 2876 a ndisko 92.8 9.0 © ato ndieicl 21.9 0.8 2028 0 edo 9.0 0.0 ° a dike 0.0 2.0 ° ° This example shows the output of an sostat command run that is updated every second (interval=1) and generated only two reports (count = 2). Each report is a combination of a TTY and CPU utilization report and a disk Utilization report. This example shows a system with five hard disks (hdisk0-hdisk4) and one CD-ROM drive. The first report shows the summary statistics since system startup, providing the collective summary of /O operations on each drive. From the previous example, you can determine that, hdisk1 has been the most actively used drive. The second report provided is the actual drive usage during a one second interval, During the report, there was a copy command started to provide disk VO activity. Chapter 4. Disk VO performance monitoring tools 1094.2.1 Historical disk VO 110 In AIX Version 4.3 the system does not collect a history of disk activity by default, as some system resources are consumed for this operation. The system administrator has to decide whether to activate the disk I/O history or not. Note If the disk /O history is disabled, the iostat command displays a message similar to the following: Collecting disk 1/0 history is an AIX operating system setting that can be enabled or disabled in SMIT using: anit chgsys. Figure 14 shows the corresponding SMIT screen, Change 7 Show Characteristics of Operating Syste Hype or select values in entry Fields. Press Enter AFTER waking all deoired changes. entry Fieldod Noximun sunber of PROCESSES alloved per veer £228 Noxinin tuner of pages in block 1/0 BUFFER CACHE (20) Noxinin Wogtes of Fesl. memory alloved for MURS. (02 Fitonat scaly REBOOT system after aeresh iy : HIGH ater work for pendang urate I/0s per file (07 of LOW water mark for pending rite I/Os per File (0) oy fiaount of Usable phiysteal mower) in Kjos 524288 Enabie Full CORE dup else . Ue pren20 Style CORE dump alse : PU Bosra Alsable : F2Cancel FosConand Feat Floste Enter: Figure 14. SMIT screen for changing characteristics of operating system When the historical disk I/O is activated, ignore the first report if you are looking for real-time behavior of your system. IBM Certiication Study Guide AIX Performance and System Tuning4.2.2 TTY and CPU utilization report The first report section displayed by the iostat command contains the TTY and CPU utilization report, The following columns are displayed and their meanings provided: tin ‘Shows the total characters per second read by all TTY devices. tout Indicates the total characters per second written to all TTY devices. %euser Shows the percentage of CPU utilization that occurred while executing at the user level (application). % sys Shows the percentage of CPU utilization that occurred while executing at the system level (kernel). % idle Shows the percentage of time that the CPU or CPUs were idle while the system did not have an outstanding disk /O request. % iowait Shows the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request. The TTY information columns, tin and tout, show the number of characters read and written by all TTY devices. This includes both real and pseudo TTY devices. Real TTY devices are those connected to an asynchronous port, such as serial terminals, modems, FAX, and so on. Pseudo TTY devices are sessions and xterm (or other X based terminal emulators, such as cern). elm atcern and Since the processing of character /O consumes CPU resources, it is important to monitor the relation between increased TTY activity and CPU utilization. If such a relationship exists, the TTY devices, along with the applications using these TTY devices, should be analyzed. For example, a FAX application could be improved by enhancing the speed of the TTY port parameters so that a file transfer would become faster and more efficient. More information about TTY performance, especially when your system has multiport adapters (8-, 16-, or 64-port adapters) can be found in AIX Versions 3.2 and 4 Performance Tuning Guide, SC23-2365, in the chapter named Tuning Asynchronous Connections for High-Speed Transfers. The CPU statistics columns, % user, % sys, % idle, and % iowait, provide information about the CPU usage. The same information is also reported in the vneeat command output in the columns us, sy, id, and wa. In general, a high % iowait indicates that the system has a memory shortage, due to paging or an inefficient /O subsystem configuration. Understanding CChapter 4. Disk VO performance monitoring tools 111