Professional Documents
Culture Documents
Oracle OS Watcher
Oracle OS Watcher
Troubleshooting?
Oracle OS Watcher (OSWatcher) is a tool to help Remote DBA's to trouble shoot
Database performance, Cluster reboot, node eviction, DB server reboot, DB instance
Crash related issues and many more.
Now, DBA can get Database related stats from AWR reports but not OS related stats for
last day, to overcome this challenge Oracle introduce OS Watcher utility, which collects
OS stats data at a frequency of five minutes and keep it for seven days (default
settings). So Now, DBA need not to worry about historical OS stats.
To Troubleshoot Database performance related issues AWR, ADDM and OS Watcher logs
are the first place to start for a Remote DBA. Whereas for Cluster reboot, node eviction,
DB server reboot Alter log files, OS Watcher and System messages
(/var/log/messages) plays an important role.
1. Download tar file from Oracle Support Article "OSWatcher Black Box (Includes:
[Video]) [ID 301137.1]"
2. Copy the file oswbb601.tar to the directory where oswbb is to be installed.
3. Extract tar file with oracle user
# tar xvf oswbb601.tar
Example 1: ./startOSW.sh 60 10
This would start the tool and collect data at 60 second intervals and log the last 10 hours
of data to archive files.
Example 2: ./startOSW.sh
This would use the default values of 30, 48 and collect data at 30 second intervals and
log the last 48 hours of data to archive files.
Example 3: ./startOSW.sh 20 24 gzip
This would start the tool and collect data at 20 second intervals and log the last 24 hours
of data to archive files. Each file would be compressed by running the gzip utility after
creation.
STOPPING OSW:
To stop the OSW utility, execute the stopOSW.sh command. This terminates all the
processes associated with the tool.
Example: ./stopOSW.sh
The default location of OS Watcher files is /opt/oracle.oswatcher/osw/archive. To
collect OS Watcher files for a particular day use below command.
# cd /opt/oracle.oswatcher/osw/archive
# find . -name '*13.03.15*' -print -exec zip /tmp/osw_`hostname`.zip {} \;
Recently, Remote DBA face a node eviction issue in a three node Real Application Cluster
environment. To resolve this, we start looking at alter log files for DB and RAC env and
OS Watcher logs. In OsWatcher Mpstat values at time of issue are given below
CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
16:27:00 all 2.60 0.00 1.64 46.53 0.01 0.06 0.00 49.16 3088.40
16:27:05 all 0.44 0.00 1.50 17.50 0.01 0.05 0.00 80.50 2397.39
16:27:10 all 0.47 0.00 0.62 12.98 0.02 0.03 0.00 85.88 2361.48
16:27:15 all 1.00 0.00 14.08 5.34 0.01 0.04 0.00 79.52 2097.98
16:27:21 all 1.11 0.00 72.81 25.22 0.02 0.23 0.00 0.61 6164.79
16:27:28 all 0.73 0.00 98.59 0.56 0.02 0.10 0.00 0.00 5348.05
16:27:33 all 0.39 0.00 99.44 0.11 0.02 0.04 0.00 0.00 3578.19
16:30:02 all 0.16 0.00 96.24 2.63 0.00 0.03 0.00 0.93 1688.58
16:30:07 all 1.34 0.00 1.79 60.13 0.02 0.09 0.00 36.62 5086.03
16:30:12 all 0.99 0.00 0.49 80.87 0.03 0.07 0.00 17.56 3650.30
From the above data, this is clear that, All CUP were 100% utilized which cause system
resource at crunch and system was rebooted. Now, DBA needs to look what case this
high system utilization.
To troubleshoot this, DBA check top output at time of issue from OS Watcher logs in the
top folder. There were around 200 Parallel process running at time. Then I cross check
these process with another Top command output at good time, and it was clear that,
these process was not running at good time. In conclusion, high number of parallel
processes cause this issue.
So, the problem and reason is clear with the help of Oracle OS Watcher tool. This is a
simple real life scenario to understand how OS watcher can help remote DBA to resolve
issues. I have also mentioned steps for detailed analysis of OS Watcher logs.
1. Read or Analyze oswiostat Outpu t: These files will contain output from the 'iostat'
command that is obtained and archive by OS Watcher on hourly basis (default). These
files will only exist if 'iostat' is installed on the OS and if the osw user has privileges to
run the utility.
The iostat command is used for monitoring system input/output device loading by
observing the time the physical disks and their average data transfer rate. Below are two
example of OSWiostat.
Time: 00:01:09
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 10.24 101.92 10.28 29.60 569.53 1057.09 40.79 0.21 5.30 0.53 2.11
sda1 0.00 0.00 0.00 0.00 0.17 0.00 138.66 0.00 12.37 3.45 0.00
sda2 10.24 101.92 10.28 29.57 569.36 1057.09 40.81 0.21 5.30 0.53 2.11
dm-0 0.00 0.00 1.72 77.98 75.95 623.85 8.78 1.20 14.99 0.08 0.67
dm-1 0.00 0.00 0.46 2.37 3.80 18.94 8.04 0.01 2.71 0.29 0.08
dm-2 0.00 0.00 7.44 50.74 278.30 410.79 11.84 0.72 12.30 0.23 1.33
dm-3 0.00 0.00 0.00 0.00 0.15 0.00 509.61 0.00 46.78 7.53 0.00
dm-4 0.00 0.00 0.49 0.00 117.41 0.02 238.95 0.00 1.94 1.05 0.05
dm-5 0.00 0.00 0.05 0.00 10.84 0.00 230.78 0.00 2.58 1.34 0.01
dm-6 0.00 0.00 0.00 0.00 0.10 0.00 479.96 0.00 54.94 8.70 0.00
Time: 07:32:57
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 163.40 7.73 2074.74 53.95 73642.61 493.47 34.83 107.13 50.07 0.47 100.07
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 163.40 7.73 2074.74 53.95 73642.61 493.47 34.83 107.13 50.07 0.47 100.07
dm-0 0.00 0.00 201.03 0.86 8412.37 6.87 41.70 58.68 281.80 4.96 100.07
dm-1 0.00 0.00 180.76 26.46 1446.05 211.68 8.00 25.24 119.01 4.83 100.07
dm-2 0.00 0.00 1868.90 34.54 63913.40 276.29 33.72 332.23 172.22 0.53 100.09
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
In this stat Remote DBA has to look at read and write happening into the system. Sudden
change in read write pattern of system means some major change has done into the
system.
Observation: - These are two iostats output shown above from a system having good
performance at 12 oclock (Time: 00:01:09) and dull at 7 O'clock (Time: 07:32:57). We
can easily judge, that few disks sda, sda2, dm-0, dm-1 and dm-2 is 100% busy (%util)
and has very high read per sec (r/s) in above screen shot. Which clearly show that,
system is performing high no of read operations.
This is also evident from "avg-cpu" stats, at good time system is 92% idle and at load
time it 0% idle.
To overcome this, DBA has to migrate data from disks sda, sda2, dm-0, dm-1 and dm-2
to dm-3, dm-4, dm-5 and dm-6 which has low read write and this will balance system
read writes.
2. Read or Analyze mpstat Output: These files will contain output from the 'mpstat'
command that is obtained and archive by OS Watcher on hourly basis (default). These
files will only exist if 'mpstat' is installed on the OS and if the osw user has privileges to
run the utility.
I prefer to start all OS Watcher related investigations from this file. These stats have a
summary of the system for database administrator. The below output is from a Linux
system.
06:13:44 CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
06:13:49 all 6.26 0.00 1.32 0.00 0.01 0.02 0.00 92.39 36448.70
06:13:54 all 8.17 0.00 1.92 0.01 0.00 0.05 0.00 89.86 38918.09
06:13:59 all 8.11 0.00 1.18 0.01 0.00 0.05 0.00 90.65 40989.86
06:14:04 all 8.04 0.00 1.25 0.06 0.00 0.05 0.00 90.61 40242.86
06:14:09 all 6.62 0.00 1.27 0.02 0.00 0.05 0.00 92.04 37460.32
06:14:14 all 7.56 0.00 1.47 0.02 0.00 0.02 0.00 90.94 37288.67
06:14:19 all 7.19 0.00 1.21 0.14 0.00 0.02 0.00 91.44 36947.91
06:14:24 all 6.50 0.00 1.02 0.01 0.00 0.02 0.00 92.45 35792.86
06:14:29 all 7.28 0.00 1.82 0.01 0.00 0.02 0.00 90.87 36795.42
06:14:34 all 7.37 0.02 1.20 0.02 0.00 0.01 0.00 91.37 36818.80
06:14:39 all 7.41 0.00 1.05 0.02 0.00 0.02 0.00 91.49 36874.90
06:14:44 all 7.15 0.01 1.62 0.04 0.00 0.02 0.00 91.16 35904.77
06:14:49 all 7.21 0.00 1.22 0.14 0.00 0.02 0.00 91.41 38867.73
06:14:54 all 7.31 0.00 1.00 0.00 0.00 0.03 0.00 91.65 39378.74
Important Field's Description: However, most of the fields has descriptive name.
CPU : "all" represents stats about all available CPU's in the system.
%user : Out of 100% CPU, How much is getting used by USER programs.
%sys : Out of 100% CPU, How much is getting used by system programs.
%iowait: Out of 100% CPU, How much is consumed by IO Waits.
%idle : After allocation CUP to all above resource, How much is still idle.
zzz ***Tue Apr 23 07:23:02 EDT 2013 Sample interval: 5 seconds
07:24:20 CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
07:24:25 all 2.74 0.00 97.16 0.00 0.00 0.10 0.00 0.00 39066.67
07:24:30 all 3.06 0.00 96.87 0.00 0.00 0.07 0.00 0.00 37637.52
07:24:37 all 3.13 0.00 96.79 0.01 0.00 0.07 0.00 0.00 36788.64
07:24:42 all 2.69 0.00 97.17 0.05 0.00 0.09 0.00 0.00 38270.04
07:24:48 all 3.86 0.01 94.92 1.02 0.00 0.20 0.00 0.00 43247.39
07:24:53 all 3.51 0.00 96.19 0.20 0.00 0.11 0.00 0.00 39887.45
07:24:59 all 4.22 0.00 93.51 2.12 0.00 0.15 0.00 0.00 40638.08
07:25:04 all 6.26 0.00 85.04 8.56 0.00 0.13 0.00 0.00 41915.79
07:25:09 all 8.69 0.00 67.31 23.85 0.00 0.11 0.00 0.03 44586.56
07:25:15 all 8.09 0.00 80.62 11.17 0.00 0.12 0.00 0.00 44321.66
07:25:21 all 7.18 0.00 71.95 20.80 0.00 0.07 0.00 0.00 35399.65
07:25:26 all 6.69 0.00 68.20 24.97 0.01 0.12 0.00 0.00 38734.99
07:25:31 all 7.11 0.01 74.71 18.09 0.00 0.08 0.00 0.00 36695.68
07:25:36 all 7.46 0.00 14.17 78.20 0.00 0.05 0.00 0.13 32934.53
07:25:50 all 9.71 0.00 23.99 66.24 0.00 0.05 0.00 0.00 33617.64
07:25:56 all 7.80 0.00 85.97 6.13 0.00 0.10 0.00 0.00 41234.83
I hope difference is clear after seeing both mpstat output good at Tue Apr 23 06:13:44
and heavy load at Tue Apr 23 07:23:02. At good time, %sys CPU utilization was 1% while
at time of load it has reached upto 97% and RAC node was rebooted after this time
due to lack of Resources. The important fields to look at are %user, %sys, %iowait
and %idle.
So, If Remote DBA observe any kind of load increase on system, He has to look at mpstat
output and then further dig out what has added load on the system. The reason could be
many like some more database sessions are introduced, a batch job has started, RMAN
backup is executing, some OS utility has introduced. We will also look into this in top
command analysis detail.
3. Read or Analyze Top Output: Top provides output of Top command on hourly basis
on Operating system. The process listed in top output are in descending order of CPU
utilization. i.e. highest CUP utilizing process first and so on.
There are many situations, when Remote DBA figured out that, CPU utilization on the
system has suddenly increased but number of process on the system are still same.
A situation:- Yesterday, there was CPU utilization spike despite of no load increase. In
these type of situation Top command is first place to start analysis for a DBA.
zzz ***Tue Apr 23 03:13:44 EDT 2013 Sample interval: 5 seconds. All measurements in KB (1024 bytes)
top - 04:13:44 up 22 days, 21:12, 10 users, load average: 65.80, 169.78, 117.65
20343 oracle 20 0 13.4g 10g 5864 R 98.4 10.7 18:56.54 oraclevntrd2 (LOCAL=NO)
30180 root 20 0 11872 2312 656 R 98.4 0.0 0:00.68 /bin/netstat -n -p -l
6568 root 39 19 0 0 0 R 89.9 0.0 263:39.04 [kipmi0]
30262 root 20 0 23704 3116 1048 R 11.9 0.0 0:00.15 /usr/bin/top -b -d 5 -n 720
4921 root RT 0 247m 86m 55m S 6.8 0.1 328:08.44 /u01/app/11.2.0.3/grid/bin/osysmond.bin
28116 oracle 20 0 2623m 71m 14m S 6.8 0.1 51:51.62 /u01/app/11.2.0.3/grid/bin/oraagent.bin
4970 grid RT 0 359m 176m 54m S 5.1 0.2 157:05.89 /u01/app/11.2.0.3/grid/bin/ocssd.bin
64 root 20 0 0 0 0 S 1.7 0.0 4:39.22 [ksoftirqd/20]
4903 root 20 0 367m 20m 13m S 1.7 0.0 26:09.97 /u01/app/11.2.0.3/grid/bin/orarootagent.bin
6496 root 20 0 1274m 15m 11m S 1.7 0.0 28:27.53 /u01/app/11.2.0.3/grid/bin/orarootagent.bin
6535 oracle 20 0 1830m 263m 4620 S 1.7 0.3 88:05.31
/u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/java -Xmx128M -server -Djava.secu
7803 oracle -2 0 1266m 11m 4068 S 1.7 0.0 9:15.42 ora_lms0_oradb2
7874 oracle -2 0 1266m 15m 4188 S 1.7 0.0 9:16.20 ora_lms0_oradb2
7999 oracle 20 0 1284m 10m 3292 S 1.7 0.0 2:49.08 ora_lmd0_oradb2
8297 oracle 20 0 1230m 3368 2864 S 1.7 0.0 0:39.95 ora_pmon_oradb2
8333 oracle -2 0 1252m 2380 2108 S 1.7 0.0 13:19.99 ora_vktm_bid2
8443 oracle -2 0 1252m 2340 2096 S 1.7 0.0 13:21.86 ora_vktm_oradb2
8535 oracle 20 0 1253m 2712 2412 S 1.7 0.0 0:14.28 ora_dskm_oradb2
8727 oracle -2 0 1266m 11m 3656 S 1.7 0.0 9:01.37 ora_lms0_im1d2
8905 oracle 20 0 1267m 13m 3468 S 1.7 0.0 9:52.75 ora_dia0_pstd2
"zzz ***Tue Apr 23 03:13:44 EDT 2013 Sample interval: 5 seconds. All measurements in
KB (1024 bytes)"
This Shows time when this stats was collected.
"top - 04:13:44 up 22 days, 21:12, 10 users"
Time with how long system has been up and running (22 days).
In this example, there were 65.80 process read to run one minute ago, 169.78 five
minutes ago and 117.65 process read to use CPU fifteen minutes before. While in below
screen shot load average is very low (load average: 2.93, 43.22, 75.56) A dramatic rise
in the load average can indicate a serious performance problem.
top - 04:20:53 up 22 days, 21:19, 10 users, load average: 2.93, 43.22, 75.56
Tasks: 2247 total, 2 running, 2181 sleeping, 0 stopped, 64 zombie
26721 oracle 20 0 1234m 15m 12m S 28.5 0.0 8:25.77 oracleoracledb2 (LOCAL=NO)
26729 oracle 20 0 1234m 15m 12m S 28.3 0.0 8:17.08 oracleoracledb2 (LOCAL=NO)
26215 oracle 20 0 1234m 13m 10m S 19.4 0.0 56:10.91 oracleoracledb2 (LOCAL=NO)
25867 oracle 20 0 1238m 17m 11m R 8.6 0.0 8:08.25 oracleoracledb2 (LOCAL=NO)
4921 root RT 0 247m 86m 55m S 7.7 0.1 328:43.44 /u01/app/11.2.0.3/grid/bin/osysmond.bin
26706 oracle 20 0 1235m 16m 13m S 6.3 0.0 8:00.49 oracleoracledb2 (LOCAL=NO)
28116 oracle 20 0 2623m 71m 14m S 5.7 0.1 52:15.90 /u01/app/11.2.0.3/grid/bin/oraagent.bin
26710 oracle 20 0 1235m 15m 12m S 3.7 0.0 8:06.62 oracleoracledb2 (LOCAL=NO)
4970 grid RT 0 359m 176m 54m S 3.3 0.2 157:20.60 /u01/app/11.2.0.3/grid/bin/ocssd.bin
26708 oracle 20 0 1235m 17m 13m S 2.4 0.0 8:09.64 oracleoracledb2 (LOCAL=NO)
30262 root 20 0 23712 3228 1128 R 1.8 0.0 0:08.14 /usr/bin/top -b -d 5 -n 720
26719 oracle 20 0 1235m 16m 13m S 1.2 0.0 8:33.47 oracleoracledb2 (LOCAL=NO)
8399 oracle -2 0 1242m 15m 4684 S 1.0 0.0 20:43.88 ora_lms0_oracledb2
Above information shows no. of CPU's into the system and each CUP utilization. CPU's
are named as cpu0, cpu1 and cpu2, Here, this system has 12 CPU's and each CPU
utilization is shown above.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26721 oracle 20 0 1234m 15m 12m S 28.5 0.0 8:25.77 oracleoracledb2 (LOCAL=NO)
26729 oracle 20 0 1234m 15m 12m S 28.3 0.0 8:17.08 oracleoracledb2 (LOCAL=NO)
26215 oracle 20 0 1234m 13m 10m S 19.4 0.0 56:10.91 oracleoracledb2 (LOCAL=NO)
25867 oracle 20 0 1238m 17m 11m R 8.6 0.0 8:08.25 oracleoracledb2 (LOCAL=NO)
This list shows the number of process and their details running at time of snapshot taken
by OS Watcher. Important fields to look at are
However, there are few most stats available in OS Watcher to look at, but after looking at
above three stats can give you a fair idea about load on the system and DBA can start
investigations accordingly.
In conclusion, I would say if you have identified the load increase on the system with
OS Watcher analysis that is great, but if there was no load increase on the system at all,
then this is not end of the world, you have next tool available for analyze why Database
is performing slow i.e. AWR Report, Generate an awr report and start reading or
analyzing AWR report to look at actual cause of the issue.
So before generating the AWR Report let's find time interval for which we need AWR
Report.
Suppose we need AWR Report for 05-MAY-11. To achieve this, we need snapshot ID for
this data.
How to find Snap Id: Select from dba_hist_snapshot view which have two fields
"BEGIN_INTERVAL_TIME" and "END_INTERVAL_TIME". These fields give exact value for
time this snap shot belongs.
From above table select your snap_id in the meantime for 05-MAY-11. In this case
starting snap_id is 2676 and it goes up to 2699.
1. Log in to database:
[oracle@test ~]$ export ORACLE_SID=test01
[oracle@test ~]$ sqlplus sys as sysdba
3. Choose Report Type: It will prompt for AWR report type. This report can be generated
in two types "HTML" and "TEXT". Default is HTML
4. Snap details:You can give any value it will show you all snap id's and detail for given
no of days.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap:2676
Enter value for end_snap: 2699
Here you need to give begin and end snap id.
[oracle@test]$ pwd
/home/oracle
AWR stands for Automatically workload repository, Though there could be many types of database
performance issues, but when whole database is slow, then there are two possibilities.
2. Stick to Particular Time: "Database is performing slow" will not help anymore to
resolve performace issues. We have to have a specific time like Database was slow yesterday at 1 Pm
and continue till 4Pm. Here, DBA will get a report for these three hours.
3. Split Large AWR Report into Smaller Reports: Instead of having one report
for long time like one report for 4hrs. it's is better to have four reports each for one hour. This will help
to isolate the problem.
In case of RAC env. generate one report for each instance. Once, you have generated AWR report.
Now, it's time of analyze the report. Since, AWR report is a huge report and area to look into AWR is
also depends on problem to problem. Here, I am list most common area for a DBA to look into which
will give a clear picture of the issue.
2. Host Configuration:
This will give you name, platform CUP, socket and RAM etc. Important thing to notice
is number of cores into the system. In this example there are 12 CUP's in Cores.
You can find, DB time is very large as compared to Elapse time, which is not a concern.
Check if you have taken a report for the time having performance problem. If yes fine,
other wise take a report for performance problem time.
4. Load Profile:
Here are few important stats for a DBA to look into. Fist is "DB CPU(s)" per second.
Before that let's understand how DB CUP's work. Suppose you have 12 cores into the
system. So, per wall clock second you have 12 seconds to work on CPU.
So, if "DB CPU(s)" per second in this report > cores in (Host Configuration (#2)).
means env is CPU bound and either need more CPU's or need to further check is this
happening all the time or just for a fraction of time. As per my experience there are very
few cases, when system is CPU bound.
In this case, machine has 12 cores and DB CPU(s) per second is 6.8. So, this is
not a CPU bound case.
Next stat to look at are Parses and Hard parses. If the ratio of hard parse to parse is
high, this means Database is performing more hard parse. So, needs to look at
parameters like cursor_sharing and application level for bind variables etc.
Here, first of all check for wait class if wait class is User I/O , System I/O, Others etc
this could be fine but if wait class has value "Concurrency" then there could be some
serious problem. Next to look at is Time (s) which show how many times DB was waiting
in this class and then Avg Wait (ms). If Time(s) are high but Avg Wait (ms) is low
then you can ignore this. If both are high or Avg Wait (ms) is high then this has to
further investigate.
In the above screen shot, most of the resource are taken by DB CPU = 64% DB time.
Taking resource by DB CUP is a normal situation.
Let's take an example, In which event is "log file switch (checkpoint incomplete) " which
has high waits, huge Time (s) and large values in Avg Wait (ms) and wait class is
configuration. So, here you have to investigate and resolve log file switch (checkpoint
incomplete).
Host CPU, Instance CPU and Memory Statistics are self explanatory. Next is RAC
Statistics, I did not find any issue in these stats most of the time.
Because this is cumulative time i.e. In this case SQL execute elapsed time is taking 89%
of DB time, which includes it sub parts like parse time elapsed, hard parse elapsed time
etc. So, if you find Hard parse time elapsed is taking more %. So investigate further so
on and so forth.
This report shows, system is 62 and 70% idle at time of report taken, So, there is no
resource crunch at system level. But if, you found very high busy, user or sys % and
indeed this will led to low idle %. Investigate what is causing this. OS Watcher is the tool
which can help in this direction.
Next, very crucial part of AWR report for a DBA is SQL Statistics. Which has all sql query
details executed during report time interval.
We will explore few of them, To understand, how to analyzed these reports. Let's start
with
In this report, look for query has low executions and high Elapsed time per Exec (s)
and this query could be a candidate for troubleshooting or optimizations. In above
report, you can see first query has maximum Elapsed time but no execution. So you
have to investigate this.
In Important point, if executions is 0, it doesn't means query is not executing, this might
be the case when query was still executing and you took AWR report. That's why query
completion was not covered in Report.
From above stat, look for queries using highest CPU Times, If a query shows
executions 0, this doesn't means query is not executing. It might be same case as in
SQL queries ordered by Elapsed time. The query is still executing and you have taken the
snapshot.
However, There are so many other stats in AWR Report which a DBA needs to consider, I
have listed only ten of them but these are the most commonly used stats for any
performance related information.
Please share you view about this article, Does it helps you to understand, How to analyze
AWR Report.