A Study of the Top 15 Wait Events that Cause Database Delays

As part of Confio Software’s consulting and database tuning exercises, we have

the chance to observe performance data from hundreds of customers with
databases across thousands of applications. This statistical analysis looks at the
source of database delays as experienced across a range of different scenarios.
This research is aimed to help companies improve their database and application
performance by helping them focus on the most serious causes of database
related application wait time.

Background

This study is a comprehensive analysis by Confio Software across hundreds of

production databases in multiple industries as measured by wait events. As the
Oracle database processes SQL requests, there are hundreds of separate processes
in the Oracle kernel through which the SQL statement passes. These are known as
“wait events” because they impose a delay as the SQL waits for processing. Oracle
and other database vendors have identified these and exposed them for analysis.

In Oracle 10g, there are over 800 different wait events. Because these wait events
represent shared resources, where the SQLs must wait for their turn to be
processed, they deliver an excellent method of identifying where delays accumulate
in executing SQL requests, and exposing what are the bottlenecks limiting database
performance for application users. In recent years, leading analysts and experts
have identified wait event analysis as the emerging best practice for database
tuning, based on the ability to isolate root cause and to make improvements that
directly impact end-users. For more detail, see “Oracle Wait Interface, © Oracle
Press 2004” (1) as an excellent reference. 1

Confio Ignite for Oracle is a performance monitoring and tuning tool that focuses on
measuring wait time as the most important determinant of database performance.
Built specifically to monitor and analyze wait events on a continuous basis, Ignite
utilizes an agentless architecture to enable capture of detailed wait time data without
imposing measurable load on the monitored server.

Study Results

The databases in the study focus on Oracle implementations, the most common
among the Oracle, Microsoft and IBM databases supported by Confio. Highlights of
the research include:
• Two-thirds of all wait time accumulates in Input/Output (I/O) related delays
• Cache requests related to clustering are the next most common source of
wait-time in Oracle RAC databases, and can be a significant bottleneck
• The top 15 wait events ranked by total wait time represent over 90% of all
wait events collected

1
Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning. R. Shee, K. Deshpande,
K. Gopalakrishnan. Oracle Press 2004.

May 2007 Confio Software www.confio.com

Along with an index and ranking of the top 15 wait events ranked by total wait time,
this study also provides several high level suggestions for reducing the wait time that
will result in increased application performance.

Wait Event / Suggestions Wait Class Pct Seconds

db file sequential read User I/O 28% 81,907,388
tune indexing
tune SQL
tune disks
increase buffer cache
db file scattered read User I/O 27% 76,886,866
add indexes
tune SQL
tune disks
refresh statistics
create materialized view
direct path read / direct path read temp User I/O 10% 16,699,540
review P1 (file id), P2 (starting dba), P3 (blocks)
if reading temporary data
o increase pga_aggregate_target
(workarea_size_policy=AUTO)
o increase sort_area_size
(workarea_size_policy<>AUTO)
if reading application data
o could be related to parallel query
o ensure DISK_ASYNC_IO = TRUE
cache temporary datafiles at O/S level
global cache cr request Cluster 5% 14,390,938
RAC event similar to buffer busy waits
tune SQL to request less data
tune network latency between RAC nodes
localize data access
buffer busy waits / read by other session Concurrency 5% 11,378,165
tune SQL
tune indexing
we often see this event along with full table scans
review P1 (file id), P2 (block id) for hot blocks
if the SQL is inserting data, consider increasing
FREELISTS and/or INITRANS
if the waits are on segment header blocks, consider
increasing extent sizes
SQL*Net more data from dblink Network 4% 10,497,202
may not be a problem
reduce amount of data transferred across dblink
tune network between databases
log file sync Commit 3% 10,039,584
tune applications to commit less often
tune disks where redo logs exist
try using nologging / unrecoverable options
log buffer could be too large
direct path write / direct path write temp User I/O 3% 7,371,823
review P1 (file id), P2 (starting dba), P3 (blocks)
similar approaches as “direct path read”

May 2007 Confio Software www.confio.com

 could be related to direct path loads
library cache lock Concurrency 2% 6,086,052
need to find the session holding the lock
look for DML manipulating an object being accessed
if the session is trying to recompile PL/SQL, look for
other sessions executing the code
Review Metalink Note: 122793.1 for other ideas
SQL*Net more data to client Network 2% 5,490,140
may not be a problem
reduce amount of data being returned by query
tune network access between client and database
db file parallel read / db file parallel write User I/O 2% 5,180,632
tune SQL
tune indexing
tune disk I/O
increase buffer cache
library cache pin Concurrency 1% 3,336,422
if many sessions are waiting, tune shared pool
if few sessions are waiting, lock is session specific
o find the blocking sessions
o review Metalink Note: 115656.1 for ideas
log buffer space Configuration 1% 1,900,724
increase LOG_BUFFER parameter
move log files to faster disks
tune application
o use NOLOGGING
o look for poor behavior that updates an
entire row when only a few columns change
enq: TX - row lock contention (10g) / enqueue (9i) Application 1% 865,463
multiple sessions are updating same row
cache buffers chains Concurrency 1% 722,427
typically caused by hot blocks
distribute data activity

Conclusions

As would be expected, when combining statistics across all customer database

applications, I/O operations create the most wait time. The study captured nearly 6
aggregate years of wait time in I/O operations alone, demonstrating that this can
have a very significant effect on end users. I/O can be tuned through an
understanding of the specific SQLs performing the operations, the type of I/O, and
the application code that generates the SQL statement. All of these are visible with
a fine degree of granularity using wait event analysis.

For individual databases, the less frequently encountered wait events will often be
the most critical hidden problem. In a specific database situation, wait events like
buffer busy, library cache lock, cache buffers, or enqueue can be the most significant
source of delay. Because these issues are much less common than typical I/O, they
are harder to pinpoint and harder to resolve. Wait event tools will show the exact
source of the bottleneck and where to begin resolving it.

May 2007 Confio Software www.confio.com

In general, if a skilled DBA knows the root cause of the problem, he/she can resolve
it and have a significant positive impact on user performance. The challenge is
finding definitive evidence of where the problem lies – which database processes and
for which SQL statement. That is the value of wait event analysis.

Confio provides performance consulting and hands-on training with every free trial.
The objective is to make the DBA skilled in using the powerful information accessible
through wait event analysis to find and resolve problems on his/her own.

Confio Igniter Suite Free Trial and Consultation

Confio Igniter Suite is the leading wait-time based performance tool for Oracle, DB2,
SQL Server and J2EE based applications. Database monitoring installs in 30
minutes, without any agent software on the monitored database server. Free trials
are available via download from www.confio.com. Each trial includes free
performance consultation with Confio expert DBAs, where the actual performance
bottlenecks and tuning recommendations are presented.

May 2007 Confio Software www.confio.com