Healthcheck Detect Diagnose Resolve

Oracle Health Check
Table of Contents
Table of Contents
Section One: About Quest Software
Section Two: Database Tuning Methodology: Detect, Diagnose, Resolve

• Overview of Quest’s Detect, Diagnose, Resolve Methodology
• Foglight® Overview
• Quest Central™ for Oracle Overview
o Performance Diagnostics Components
o Database Administration Component
o SQL Tuning Component
o Space Management Component
Section Three: Sample Report—Excerpt of Health Check done in September 2002

• Report Cover Page
• Health Check Overview
• Sample Executive Summary
• Detailed Advice
Oracle Health Check
About Quest Software
Why Application Management?

Businesses today rely on enterprise applications for manufacturing, financials, supply chain
management, human resources, and customer relationship management—virtually every
vital function within a business. As the reliance on these applications continues to grow,
performance and availability become paramount. Simply put, your business is only as good
as your application infrastructure. At Quest Software, we understand this better than
anyone else—that is why we are 100% focused on making your application infrastructure
better.
Quest for Application Confidence
Quest Software, the leader in application management, gives you confidence in your critical
application infrastructures. This confidence is delivered in the form of reliable software
products that help you develop, deploy, manage and maintain enterprise applications
without expensive downtime or business interruption. We equip IT professionals with
software tools that are easy to use and embedded with the knowledge of our world-
renowned technical experts to automate daily tasks and simplify complex ones. As a result,
your IT organization will be able to proactively prevent performance problems and
downtime, as well as react quickly to the unforeseen—making them more productive,
capable and confident. The bottom line is a lower total cost of ownership for your application
infrastructure.
Quest Software will help you reduce costly planned and unplanned downtime, get better
performance from existing infrastructures (without throwing money at expensive hardware
upgrades), and equip your staff to do more with less. No other software provider offers a
more comprehensive approach to application management.
Oracle Health Check
Tuning Methodology: Detect, Diagnose, Resolve
Quest Software’s Performance Tuning Methodology:

Detect, Diagnose, Resolve
As an application monitor, Foglight® provides the “detect” portion of this methodology.
Quest Central™ for Oracle, through Performance Diagnostics, Database Administration, SQL
Tuning and Space Management, finishes the job by providing efficient ways to diagnose and
resolve any issues you may have in your Oracle environment.
Detect
Foglight provides 24x7 unattended monitoring for complex Oracle database environments.
With Foglight, administrators can proactively identify potential database bottlenecks before
they affect database and application performance, ensuring optimal performance and
availability for users. Foglight is designed to provide proactive notifications of impending or
actual problems and historical information. The true power behind a monitoring solution is
to provide you with notifications of impending problems or performance issues along with
notification of current issues.
Foglight also allows administrators to investigate deeper into the database. With the Oracle
Instance Overview screen, you can quickly see the percentage of CPU, cache hits and I/Os
as well as the number of Oracle sessions (user logons) for the database instance over the
specified period of time. You can investigate:
• Wait Statistics
• Tablespace Usage
• Ensure connectivity to Oracle
• Lock Contention
• Extent Failure
• Alert Logs
• Redo Logs
Foglight Notice Board Within Quest Central

The Foglight notice board within Quest Central for Oracle unites the leading solutions for
Oracle database management and the leading solution for unattended application
monitoring. The Notice Board receives alerts from Foglight to be viewed by the DBA as they
are using Quest Central. With the knowledge of an impending application problem, the DBA
can utilize Quest Central’s powerful diagnostic and resolution capabilities to ensure that the
database is not contributing to the problem.
Diagnose
Foglight’s powerful integrated diagnostic capabilities allow administrators to drill down using
Quest Central’s Performance Diagnostics Solution, Spotlight®, in near-real time. With this
combination, you can quickly identify the performance issue and provide expert advice to
resolve it.
Resolve
Quest Central’s Database Administration, SQL Tuning and Space Management components
provide deep functionality for resolving issues that Foglight and Spotlight detect and
diagnose. Covering virtually every issue DBAs face, these modules will increase your
productivity while improving system performance.
Oracle Health Check
Foglight Overview
Foglight Overview
Foglight is a unique solution that monitors every component affecting application
performance—end user, Web server, application, operating system, database, disk and
network—alerting system administrators to quickly identify actual or potential application
problems and correct these performance issues before end users are affected.
In order to effectively manage your application infrastructure, you need to collect vital
application information and correlate it across the complex IT infrastructure. Unlike
traditional monitoring solutions, Foglight is designed to be rapidly deployed to provide you
with maximum return on investment. Foglight is the only solution designed to empower you
with all the information you need to monitor your application architecture in one easy-to-
use product.
Event Notification
The true power and value behind a monitoring solution is the ability to be notified of
impending problems before they affect your business and your end users. Foglight provides
proactive alerts with pre-configured best practice notifications to aid in deployment. It also
provides you with the ability to define your own rules without learning complex proprietary
languages—allowing you to set performance thresholds and maintain service levels.
Foglight also has the power to do cross-host event correlation, allowing you to define
specific events related to your business processes. For example, an administrator can define
the CPU utilization of three Web servers as a single event. If all three servers are utilized by
more than 80%, Foglight will send a single notification. Events can also be defined based on
your specific business applications.
Intuitive User Interface Predefine actions

Foglight offers an easy-to-use graphical interface designed to make to aid in the
you smarter with the ability to easily drill down to aid in diagnosing resolution of
the root-cause of a problem. Foglight also provides detailed event problems
information and views, tightly correlated to effortlessly guide you
along the best path to fast and easy problem resolution. Define blackout
periods to disable
A powerful Web-enabled interface allows for remote administration notification for
and offers an overview of the overall health of the system from one planned events
single screen.
Data Repository for Point-in-time Analysis

The Foglight data repository is a high-performance, low maintenance database that allows
the administrator to collect and store vital application performance information. Its data can
then be extracted to perform root-cause analysis during problem resolution. Over time, it
becomes an invaluable source of historical information for point-in-time analysis, planning
of system capacity requirements and trend analysis over any period of time.
Oracle Health Check
Foglight Overview
Powerful Reporting
Foglight has hundreds of pre-defined, built-in reports based on all the graphical views.
Foglight’s integrated reporting wizard will allow the user to create custom reports from any
metric that is collected and stored in the data repository.
With Foglight, users can generate reports showing utilization or service levels based on
system, database, Web, network and application metrics over user-specified time ranges.
Its versatile report schedule allows users to easily define and schedule simple and complex
reports for viewing and distribution through e-mail or the Web-based interface.
Maintaining Service Level Agreements Schedule reports

Many organizations that rely heavily on their technical infrastructures for viewing,
demand strict SLAs on performance and availability from their IT distribution or
organizations and the applications that run their business. Foglight output to the Web
enables you to achieve stringent service levels reaching as high as console
99.99% availability. It provides a powerful and innovative facility that
allows users to define an SLA policy and report on service levels achieved.
Enhancing the End User Experience

The key requirement of an application monitor is to provide maximum availability and
performance for your end users. Foglight ensures the health of your vital business
applications through 24x7 proactive monitoring of your applications and its related
components. With Foglight, you can establish and deliver the best levels of service to your
end users.
Foglight offers an array of specialized cartridges focused on your e-business applications,

databases, or application servers—providing a complete monitoring solution for your
extended IT environment, including:
• Foglight Cartridge for mySAP.com™

• Foglight Cartridge for Siebel eBusiness Applications
• Foglight Cartridge for BEA WebLogic
• Foglight Cartridge for PeopleSoft
• Foglight Cartridge for Oracle E-Business Suite
• Foglight Cartridge for Databases
• Foglight Cartridge for Exchange
• Foglight for SQL Transactions
• Foglight for Response Time Measurement (RTM)
• Foglight Cartridge for Custom or Other Applications
For more information about Foglight, contact your Quest Sales Representative today.
Oracle Health Check
Overview of Quest Central for Oracle
Quest Central for Oracle:

Revolutionizing the DBA’s Job
Managing multiple databases manually or with a patchwork of unrelated, competitive tools
can make the DBA team’s job unnecessarily difficult. With the dramatic increase in size and
complexity of many companies’ database environments, Oracle DBAs are challenged to keep
up with numerous essential tasks to maintain their service levels.
Long known as the leader in Oracle database management tools, Quest Software has
brought its industry-leading solutions into a single, comprehensive solution: Quest Central
for Oracle. Popular products such as Spotlight on Oracle, SQLab Vision™, and Space
Manager™ have been integrated with powerful new capabilities including database
administration and database analysis to create the tool that DBAs can use all day. This
product has been designed with the DBA in mind by addressing your most pressing
requirements.
Quest Central enables you to centralize everything you need to do on a daily basis—across
all your databases—to become more effective, improve your system’s performance, and
increase availability.
Oracle Health Check
Components of Quest Central for Oracle
Performance Diagnostics Components:

Spotlight for Oracle and Database Analysis
Pinpoint the Origin of Performance Issues
In Quest Central for Oracle, Performance Diagnostics consists of two elements: the real-
time graphical displays of Spotlight on Oracle, and the in-depth analysis of your system’s
performance metrics in the Database Analysis feature.
Spotlight on Oracle, the industry’s best-known graphical diagnostics tool, improves staff
efficiency for troubleshooting, speeds time to resolution, and reduces end-user downtime by
proactively discovering performance issues in real time. Spotlight on Oracle:
• Provides an intuitive, architecturally correct GUI of process flows within an Oracle

instance
• Identifies bottlenecks using data flows and visual icons
• Displays the details of problem areas, including top sessions, inefficient SQL, locks,
latches, wait events and disk IO, helping you identify performance problems quickly
• Alerts you with an audio or visual alarm if any Oracle component is forming a
bottleneck
With Quest Central’s Database Analysis feature, it’s easy to analyze a variety of metrics
within your Oracle instance for an accurate picture of overall database health. Quest
Central’s performance analysis displays recommendations and provides advice to solve your
tuning issues. Database analysis:
• Analyzes the database and provides explicit tuning advice
• Provides historical reports, both online and in HTML, showing detailed user activity
and more
• Continues the workflow from diagnosis to resolution with embedded integration to
other components within Quest Central including SQL Tuning, Database
Administration and Space Management
The database analysis function of Quest Central is like an in-house Oracle tuning expert.
Whenever there is doubt about the performance of your Oracle database, ask your tuning
expert—Quest Central—to evaluate what to tune and how to tune it for the biggest
improvement.
Oracle Health Check
Database Administration Component

Simplify Daily DBA Tasks
Quest Central for Oracle’s Database Administration simplifies the task of maintaining
database objects with a comprehensive, easy-to-use graphical object management
interface. It allows you to quickly create, drop or alter database objects to accommodate
business and application needs. This component also ensures successful script execution
with sophisticated error-checking capabilities, and enables the DBA to schedule the script to
execute at appropriate times. You can:
• Quickly create, drop or alter database objects

• Manage Oracle security, users, roles and privileges
• Automatically build SQL scripts to execute alteration change requests
Another important aspect of database administration is managing security and permissions.

DBAs can view and control permissions from the object or user levels, ensuring that
application data is only available to authorized users. The Database Administration
component is fully integrated with other Quest Central components to streamline workflow.
Oracle Health Check
SQL Tuning Component, previously SQLab Vision

Quickly and Efficiently Optimize SQL for Faster Response Time
The SQL Tuning Component integrates the new generation of SQL tuning into Quest
Central for Oracle. Previously known industry-wide as the proven, leading-edge tuning
solution SQLab Vision, Quest Central’s SQL Tuning offers a powerful Oracle tuning
environment, non-intrusive collection technology, impact analysis capabilities and expert
tuning advice.
Quest Central's SQL Tuning solution features Quest’s renowned non-intrusive, agent based
collection technology to collect SQL statements and performance indicators from Oracle
databases and host operating systems at sub-second intervals. This solution collects data
non-intrusively, with no added overhead on the target database and minimal overhead on
the host system. StealthCollect runs 24x7, enabling you to identify and resolve problems in
the past and collect performance information automatically, requiring no management time
or active maintenance. The SQL Tuning Component:
• Captures all SQL statements and performance metrics non-intrusively at sub-second

intervals, with no overhead to Oracle
• Identifies the root of most performance problems by detecting offensive SQL
statements
• Provides immediate troubleshooting in real time, even when Oracle is overloaded or
not responding
• Eliminates performance slowdowns caused by incompatibilities between application
code and database structures
• Supports SQL tuning by both developers and DBAs with easy-to-understand explain
plans, flowcharts, pie charts and graphs
• Automatically rewrites SQL and selects the best alternative to maximize performance
• Offers expert tuning advice developed by Quest Software’s world-renowned Oracle
experts
• Demonstrates the impact of adding an index throughout the application
• Supplies an Oracle repository for capturing unlimited SQL history with detail
• Provides SQL, user, and program for ERP transactions
Quest Central for Oracle integrates Quest Software’s industry-leading Oracle tools that have
been successfully used for years. SQLab Vision is now fully integrated into this revolutionary
suite, simplifying the DBA’s daily tasks.
Oracle Health Check
Space Management Component, previously Space Manager

Improve Performance and Reclaim Disk Space
As database environments become more complex and varied, incidences of poor response
time in critical systems—even downtime—have grown exponentially in Oracle environments.
Fragmentation, excessive extents, and other space-related conditions increasingly interfere
with business continuity.
Quest Central’s Space Management module simplifies time-consuming, error-prone methods

when DBA teams need it the most. Offering complete reorganization, relocation and
restructuring for present performance and future growth, Space Management:
• Provides fast reorganizations and restructuring both inside the database and via the
file system using an intuitive GUI
• Predicts out of space conditions and offers detailed “reorg need”
• Calculates the optimum reorganization strategy
• Reorganizes multiple objects simultaneously
• Offers comprehensive capacity planning and thorough object sizing
• Supports LONG and LONG RAW data types for packaged apps
• Contains cutting-edge, patent-pending technology
Space management provides a full range of fast reorganization methods—including

reorganization for full tablespaces, individual objects, and direct repair of chained rows. To
support database restructuring, tables and indexes can be relocated easily to different
tablespaces for improved I/O distribution.
Using easy-to-read tablespace maps and GUIs, Quest Central tells you exactly which objects
will most benefit from reorganization. Detailed descriptions of exactly what is causing
problems and the anticipated benefit of the reorganization lets you maximize your efforts.
To help manage your growing database, Quest Central’s Space Management maintains a
repository of object growth statistics. This repository is automatically populated via the
periodic collection of new Oracle ANALYZE statistics.
This historical information is used to identify objects in need of reorganization and to

forecast future growth by object, tablespace or the entire database. To help avoid space-
related downtime, a sophisticated Tablespace Failure Prediction report tells you when
tablespaces will run out of space.
Oracle Health Check
Sample Report
Dear Customer:
Thank you for your interest in Quest Software’s Oracle Health Check. This binder shows you
a real-world example of the comprehensive analysis you will receive after installing Quest
Central™ for Oracle on your production system for just five days.
Enclosed, you will find the following:

• Sample Executive Summary that summarizes the findings and value of Quest
Central in your environment
• An example of a Health Check Report done in September 2002 at a
telecommunications company, including:
o Recommendations—Offers detailed advice for resolution and includes a
“confidence rating,” the level of effort you’ll expend, and the benefits your
system will receive
o Goals—Highlights broader tuning opportunities for your environment
About the Communications Company Analyzed:
Customer: Communications Provider

Environment:
• Hardware: HP-UX 9000
• Applications: Clarify and Oracle Applications (Used for Help Desk)
• Total number of databases: Over 100
• Staff: 12 Senior DBAs plus several IT consultants
This large communications provider runs a complex 24x7 environment with over 50
production databases. The DBA team and their consultants were seeking a way to easily
test and rate their trial database tools, and eventually to justify their purchase. In addition,
the Company was merging with another provider, and the team hoped for tools to ease the
integration of the two IT environments.
Through Quest Software’s Oracle Health Check program, Quest’s system consultants worked
with the Company’s DBA team for one day to install Quest Central for Oracle. Allowing
Quest Central to collect data non-intrusively for five days yielded a detailed analysis that
satisfied everyone involved: the Database Administration team received a complete
diagnosis and advice for resolution of database performance issues, and upper management
received an easy-to-read report that clarified the value of Quest Central in their
environment.
Thanks again,
Quest Software
Database Performance Health Check
The health check is a short, no-charge consulting engagement for companies who are interested in significantly improving Oracle
database performance. By non-intrusively collecting your database metrics for five days, Quest Software is able to gather dozens of
statistics and summarize them into a readable format.
Customers that have participated in Quest Software’s Oracle Health Check have realized thousands of dollars in hard savings by elimi-
nating the need for Oracle tuning consultants, and have improved database application performance and availability exponentially.
The health check process is simple, and includes the following phases:
Baseline Discovery Interview

• Environmental assessment (servers, applications, databases, storage)
• Business requirements (SLAs, types and volumes of transactions)
• Known issues (application availability, response times)
One-day Consulting Engagement for Installation and Training

• Connect to databases and collect statistical data flow information
• Use Quest’s sophisticated analysis tools to gauge overall health of environment
• Demonstrate use of Spotlight® for real-time diagnostics
Five-day Automated Data Collection

• Software automatically collects database performance metrics for five days (Does not require your time or effort)
• Email the results to your System Consultant
Receive Report from Quest Sales Representative

• The report contents include:
- Executive Summary
- Critical Issues
- Areas of Concern
- Strategic Planning
• Comprehensive recommendations include:
- Initialization Parameter Changes
- I/O Contention and Latch Contention
- SQL Tuning, Index Utilization, Buffer Cache Size
- Redo Log, Redo Buffer, Lock and Wait events
- And more
Keep the Report

The tuning advice contained within the report could double, triple, or even quadruple your system‘s performance—it has for other
participants in the program. Surveying dozens of performance metrics, the report provides diagnoses and advice in concise and easy-
to-read graphical formats. This powerful, customized tuning resource is yours to keep — still at no charge.
Quest Central™ for Oracle brings your organization incredible value. We’re so confident of the quality, return on investment and overall
value of our tools that we’re willing to offer you this unprecedented level of service.
Get Started Today!

Register for your free database health check at www.quest.com/healthcheck, or call your sales rep at 949.754.8000 for more
information.
Oracle Health Check
Sample Executive Summary
Performed at: Communications Provider (“Company”)

Dates: September 5th, 2002
Quest Software appreciates the opportunity to demonstrate the value that Quest Central for
Oracle could bring to the Company. We are so confident of the value of Quest Central that
we have installed the product in your production environment and collected key
performance metrics from your Oracle instances. This document summarizes the findings
and includes details to back up our recommendations.
Quest Central for Oracle is a comprehensive solution for managing Oracle database
environments. By providing performance diagnostics, database analysis, SQL tuning, space
management and database administration together in one integrated console, Quest Central
offers everything a DBA needs to accomplish their daily tasks.
The Database Analysis component of Quest Central for Oracle allows you to proactively
analyze the core components of your environment. Critical areas include: I/O efficiency,
data file structure, application SQL efficiency, latch and lock contention, job processes, and
tuning parameters that affect memory and processing efficiency, among others.
Comments and Findings

Quest Central’s database analysis component has generated a comprehensive report of
performance issues and solutions. The top five recommendations for increasing performance
are listed here. Details can be found in the report that follows.
Recommendations Benefit Effort

Address lock contention High Low
Transactions waiting for table locks are an indication of lock contention. 100 locks
have been observed for the SA.TABLE_DEMAND_DTL table.
Solution: Consider contention for a specific row. The application may
require that many processes update or lock the same row in the
database. Or table locks could be caused by non-indexed foreign keys.
When a non-indexed foreign key is updated, the parent table will be
subjected to a table lock until the transaction is complete.
Address high space transaction waits Medium Low
Waits for space transaction locks occur when a session is allocating or
deallocating extents.
Solution: Use true temporary tablespaces. Consider increasing INITRANS
and PCTFREE if the space reserved for transactions within a data block is
too small. This results in enqueue waits if additional transaction slots
cannot be created.
Assign dedicated temporary tablespace High Low
Make sure that every user is assigned a dedicated “true” temporary tablespace.
Solution: Use the ALTER USER command to assign true temporary
tablespace to the users listed in the recommendation.
Avoid using the order clause in sequences Medium Low
Specifying order in the create sequence command disables the sequence cache
mechanism, thus negatively affecting sequence performance.
Solution: Use the ALTER SEQUENCE command to change the storage
definitions to NOORDER of the 17 sequences found by DB Analysis.
Tune resource-intensive SQL High High

During our health check we detected a number of resource-intensive SQL. These
inefficient SQL statements cause slow response time when requesting data from
your database.
Solution: Using Quest Central you can quickly tune these resource-
intensive SQL statements with a click of a button using Xpert Advice and
AutoTune.
Oracle Health Check
Sample Executive Summary
Following are a few of the graphs included in our health check report that support our five
recommendations for increasing performance in your production environment.
Graph 1 shows that from 3:30PM to

4:15PM on July 17, 2002 there is
significant wait events for buffer busy
and latch free. During the health check,
we found that the majority of these
buffer busy waits are for data block
writes, generally caused by concurrent
inserts in a table with insufficient
freelists. Typically, a large number of
latch free waits may indicate a
bottleneck in the SGA. We determined
that the majority of these latch waits are
caused by the cache buffers chains latch
because of insufficient freelists and
setting the PCTFREE parameter too low.
Graph 1
Graph 2 shows that from 3:30PM to 6:15PM on

July 17th, 2002 there is a significant miss rate on
the buffer cache and library cache. Typically,
when the miss rate is high for the buffer cache it
means your db block buffer size is too small. If
you have a high miss rate for library cache it
either means your shared pool size is too small or
you’re not using bind variables in your SQL.
Graph 2
Graph 3 shows that from 3:30PM to

4:15PM on July 17th, 2002 there are
significant latch waits. During our health
check we determined that the cache
buffer chains latch was the primary
culprit behind these latch waits.
Contention for this latch usually
indicates that a database has very high
I/O rates. It may be possible to reduce
contention for the cache buffers by
increasing the size of the buffer cache in
your init.ora. Another possibility is to
pinpoint the hot objects using
StorageXpert and evenly distribute these
objects across your I/O subsystems to
Graph 3
reduce unnecessary I/O.
Database Analysis
Analysis Report
Instance: TELECOM.WORLD
Date Analyzed: Thursday, September 05, 2002 2:22:58 PM
Report Created: Thursday, September 05, 2002 2:47:29 PM
Recommendations:
Recommendation Confidence Effort Benefit

Address library cache latch contention 95% Medium Low
Address shared pool latch contention 21% Medium Low
Alter SQL related configuration parameters 60% High High
Assign Dedicated Temporary Tablespaces 70% Low High
Consider bind variables 95% High High
Increase cache size for nominated sequences 92% Low Medium
Lock contention high 95% Medium High
Locks waiting for tables 95% Low High
Reduce Latch free latches 95% Medium Low
Some sequences have the ORDER clause set 80% Low Medium
Space transaction waits high 95% Medium High
Tune resource-intensive SQL 95% High High
Use True Temporary Tablespaces for Users 70% Medium High
Goals:
Goal/Recommendation Confidence Effort Benefit

Exploit the Cost Based Optimizer
Optimize I/O efficiency
Optimize Redo Logs
Optimize SQL
Alter SQL related configuration parameters 60% High High
Consider bind variables 95% High High
Tune resource-intensive SQL 95% High High
Optimize sequences
Increase cache size for nominated sequences 92% Low Medium
Some sequences have the ORDER clause set 80% Low Medium
Optimize sorting efficiency
Assign Dedicated Temporary Tablespaces 70% Low High
Use True Temporary Tablespaces for Users 70% Medium High
Reduce Latch Contention
Address library cache latch contention 95% Medium Low
Address shared pool latch contention 21% Medium Low
Reduce Latch free latches 95% Medium Low
Reduce Lock Contention
Lock contention high 95% Medium High
Locks waiting for tables 95% Low High
Space transaction waits high 95% Medium High
Reduce contention
Tune the Buffer cache
Graphs:
Activity Summary
File I/O
Latch Summary
Misc reports
Wait events
This report was generated by Database Analysis.

Database Analysis
Recommendation: Lock contention high
Lock Contention
There is a high proportion of enqueue waits. This can indicate contention for the
following:
• specific rows in the database

• table locks resulting from non-indexed foreign keys
• Oracle internal locks.
The following enqueue waits have been detected:

Lock Type Gets Waits
Transaction enqueue lock 17447099 10189
Sequence number enqueue lock 117373 731
Cross-instance function invocation instance
230685 294
lock
Control file schema global enqueue lock 14470 278
Temporary segment enqueue lock or New
125652 95
block allocation enqueue lock
Cursor bind lock 2414270 12
Space transaction enqueue lock 10255 4
Being-written redo log instance lock 1492 3
Lock Contention
Common causes of excessive enqueue waits include the following:
• Contention for a specific row in the database. The application design may
require that many processes update or lock the same row in the database. For
example, when primary keys are generated using a sequence table.
• Table locks caused by un-indexed foreign keys. When an un-indexed foreign
key is updated, the parent table will be subjected to a table lock until the
transaction is complete.
• "Old-style" temporary tablespaces. When the tablespace nominated as the
temporary tablespace has not been created using the TEMPORARY clause
(introduced in Oracle 7.3) , the sessions may contend for the "space
transaction" lock.
• The space reserved for transactions within a data block is too small. When the
table or index is created, the default number of transaction slots for tables (1
slot) and indexes (2 slots) are allocated. The number of transaction slots is
determined by the INITRANS clause in the CREATE TABLE or INDEX
statement. When additional transaction slots are needed Oracle creates more
(providing there is free space in the block). However, when all of the
transaction slots are in use and there is no free space in the block, a session
requesting to lock a row in the block will encounter an enqueue wait. An
enqueue wait is encountered even when the row is not actually locked by
another process. This occurs when both PCTFREE and INITRANS are set too
low.

Database Analysis
Recommendation: Locks waiting for tables
Tables With Lock Waits

Transactions waiting for table locks are an indication of lock contention.
Lock waits have been observed for the following tables:
Table Number of observations

SA.TABLE_DEMAND_DTL 100
Lock Waits for Tables

Review each of the tables listed for possible causes of lock contention.
Some of the main causes of table locking include the following:
• Contention for a specific row in the database. The application design may
require that many processes update or lock the same row in the database. For
example, when primary keys are generated using a sequence table.
• Table locks caused by non-indexed foreign keys. When a non-indexed foreign
key is updated, the parent table will be subjected to a table lock until the
transaction is complete.

Database Analysis
Recommendation: Space transaction waits high
Reduce Space Transactions

There is a high proportion of waits for space transaction locks. Waits for space
transaction locks occur when a session is allocating or deallocating extents.
Space transaction locks can indicate the following:
• "Old-style" temporary tablespaces. When the tablespace nominated as the

temporary tablespace has not been created with the TEMPORARY clause
(introduced in ORACLE 7.3) , sessions can contend for the "space
transaction" lock.
• The space reserved for transactions within a data block is too small. When the
table or index is created, the default number of transaction slots for tables (1
slot) and indexes (2 slots) are allocated. The number of transaction slots is
determined by the INITRANS clause in the CREATE TABLE or INDEX
statement. When additional transaction slots are needed Oracle creates more
(providing there is free space in the block). However, when all of the
transaction slots are in use and there is no free space in the block, a session
requesting to lock a row in the block will encounter an enqueue wait. An
enqueue wait is encountered even when the row is not actually locked by
another process. This occurs when both PCTFREE and INITRANS are set
too low.
Locally Managed Tablespaces

In Oracle version 8i there is a new mechanism for managing space allocation in
tablespaces called locally managed tablespaces. Prior to Oracle version 8i, all
tablespaces were dictionary managed. In dictionary managed tablespaces, space
allocation is tracked in the data dictionary. This method can cause performance
bottlenecks. With locally managed tablespaces, space allocation information is
maintained in a bitmap within the tablespace itself. This management avoids
contention for the data dictionary.
Reduce Space Transaction Waits

Make sure that users are assigned a true temporary tablespace: See
recommendations for Assign Dedicated Temporary Tablespaces and Use True
Temporary Tablespaces for Users.

Database Analysis
Recommendation: Assign Dedicated Temporary Tablespaces
Use Dedicated Temporary Tablespaces

You should make sure that every user is assigned a dedicated “True” temporary
tablespace (see Use True Temporary Tablespaces for Users). Assigning a
temporary tablespace for permanent objects can cause storage fragmentation
and performance degradation.
The following table shows the list of USERS who do not have dedicated “True”
temporary tablespaces:
User Name Data Tablespace Temporary Tablespace

MCGUKEN SYSTEM SYSTEM
PRIV_QUEUE_MAP SYSTEM SYSTEM
About Temporary Tablespaces

Temporary tablespaces are used to hold temporary segments (for example, segments
used by implicit sorts to handle ORDER BY or GROUP BY clauses). A temporary
tablespace can be used only for sort segments.
It is crucial from a performance perspective that you assign dedicated temporary

tablespace to all users and schemas. There are two major reasons for this
requirement:
• Sort segments take the storage parameters from the DEFAULT STORAGE
(NEXT) clause of the temporary tablespace in which they are created. The
assignment of a dedicated temporary tablespace allows for easy modification
of default storage allocation parameters to make sure that long-running jobs
never fail when they cannot allocate enough temporary segments.
• It allows for I/O management by spreading tablespaces across different
physical disks.
It is also very important to make sure that SYSTEM tablespace is never used as a
user temporary tablespace. System tablespace is used by Oracle internally to
maintain Data Dictionary objects. Using SYSTEM tablespace can significantly degrade
the performance of your system.
Assigning Dedicated Temporary

Tablespaces
You can use this SQL script to assign dedicated temporary tablespaces to the users.
WARNING: You should test this script in a test environment. You should also back up
your database before executing any scripts generated by Database Analysis. Although
Quest Software makes every effort to ensure that these scripts are reliable,
differences in your operating environment may cause these scripts to fail. You should
also consider using Schema Manager to perform mission critical changes.
ALTER USER MCGUKEN TEMPORARY TABLESPACE

&Tablespace ;
ALTER USER PRIV_QUEUE_MAP TEMPORARY TABLESPACE
&Tablespace ;

Database Analysis
Recommendation: Use True Temporary Tablespaces for Users
Use “True” Temporary Tablespaces

Using “true” temporary tablespaces allows you to minimize the Oracle
housekeeping activities that are performed to maintain temporary segments.
“True” temporary tablespaces also optimize the efficiency of your sorts.
Database Analysis has identified users (see Assign Dedicated Temporary

Tablespaces ) who have not been assigned a “true” temporary tablespace as the
default tablespace.
“True” temporary tablespaces have been created using the CREATE TABLESPACE
command, or have been marked as temporary by an ALTER TABLESPACE
command. These tablespaces cannot hold permanent objects and are optimized for
temporary segment activity.
When temporary tablespace is not defined as “true” temporary, the effect is as follows:
• At the end of each sort operation, the temporary segment becomes a

candidate to be dropped. This space management task is performed by
SMON as a background task.
• When another sort operation requires a temporary segment, a new temporary
segment is created, possibly re-using the set of blocks that were used by the
previous sort operation.
The main problem with this approach is as follows:
• The creation of a temporary segment is a space management operation that

is performed internally by Oracle. This task imposes a lock, which can create
a bottleneck as processes line up waiting for the lock.
• Temporary segment creation also requires some data dictionary tasks to be
performed. These tasks are find the free space, register the new segment in
the data dictionary, and unregister the free space.
Creation of a designated temporary tablespace can be accomplished through the

CREATE TABLESPACE command.
The benefits of using “true” temporary tablespaces are as follows:
• Oracle guarantees that no permanent objects (tables, indexes, and so forth)

are stored in a “true” temporary tablespace.
• A temporary segment created in the tablespace is not unallocated at the end
of the sort. Instead, the extents are MARKED as free, are not placed back on
the freelist, and are retained for re-use. The Sort Extent Pool (SEP) element is
created in the SGA to maintain the temporary segments. Subsequent sort
operations are allocated extents from the temporary segments through a
memory lookup, thus avoiding most of the space management tasks that take
place when “true” temporary tablespaces are not used.
Creating “True” Temporary Tablespaces

You can use the following SQL command to create a new "true" temporary
tablespace:
CREATE TABLESPACE <tablespace name>

DATAFILE ... DEFAULT STORAGE (...) TEMPORARY;
The existing tablespaces can be turned into 'true' temporary

tablespaces by applying the following ALTER command:
ALTER TABLESPACE <tablespace name>

TEMPORARY;
Note: The DEFAULT STORAGE clause defines the size of the sort extents created
within a tablespace. It is highly recommended that this should be always a multiple of
the SORT_AREA_SIZE specified in the INIT.ORA file for your instance. The
SORT_AREA_SIZE is currently defined as 2097152 for your instance.

Database Analysis
Recommendation: Some sequences have the ORDER clause
set
Avoid Using the ORDER Clause in

Sequences
The ORDER clause causes poor performance for sequences. Do not use the
ORDER clause unless, you are using a Parallel Server, and it is absolutely
necessary.
The create sequence command also includes an ORDER clause which, if used,
guarantees that sequence numbers are generated in sequential order. This option is
intended for use with the ORACLE Parallel Server (OPS) option only. In a nonOPS
system, sequence numbers are always generated in sequential order.
Specifying order in the create sequence command disables the sequence cache
mechanism, thus negatively affecting sequence performance. You should not specify
the ORDER clause unless you are in a Parallel Server environment, and even then,
only when you absolutely require that sequence numbers be generated in sequential
order.
Since this database is not running in an OPS environment, there is no valid reason for
specifing the ORDER clause
The following sequences have the ORDER clause specified and appear to be active:
Sequence Name
SA.SEQ_MESSAGE
SA.SEQ_CONDITION
SA.SEQ_CONTACT
SA.SEQ_CONTACT_ROLE
SA.SEQ_TIME_BOMB
SA.SEQ_SITE
SA.SEQ_ACT_ENTRY
SA.SEQ_ADDRESS
SA.SEQ_INV_ROLE
SA.SEQ_DEMAND_DTL
SA.SEQ_DEMAND_HDR
SA.BASE36TBN
SA.BASE36TCL
SA.BASE36ACL
SA.BASE36ACN
SA.BASE36ACT
SA.SEQ_BUS_SITE_ROLE
When you are not in an Oracle Parallel Server (OPS) environment, change from
ORDER to NOORDER. When you are in an Oracle Parallel Server environment, you
should only use the ORDER clause when it is absolutely essential that sequence
numbers always be generated in order as generating sequence numbers in order in
an OPS environment negatively affects your system performance.
See also:
Exploiting Oracle Sequence Generators, Re:Quest, Volume 2, Issue 2
Altering Sequences
SQL script used to alter nominated sequences storage definitions.
You can use the following SQL script to alter the nominated sequences storage
definitions. You should run this script while connected as a user with the ALTER ANY
SEQUENCE privilege.
ALTER SEQUENCE SA.SEQ_MESSAGE NOORDER;
ALTER SEQUENCE SA.SEQ_CONDITION NOORDER;
ALTER SEQUENCE SA.SEQ_CONTACT NOORDER;
ALTER SEQUENCE SA.SEQ_CONTACT_ROLE NOORDER;
ALTER SEQUENCE SA.SEQ_TIME_BOMB NOORDER;
ALTER SEQUENCE SA.SEQ_SITE NOORDER;
ALTER SEQUENCE SA.SEQ_ACT_ENTRY NOORDER;
ALTER SEQUENCE SA.SEQ_ADDRESS NOORDER;
ALTER SEQUENCE SA.SEQ_INV_ROLE NOORDER;
ALTER SEQUENCE SA.SEQ_DEMAND_DTL NOORDER;

ALTER SEQUENCE SA.SEQ_DEMAND_HDR NOORDER;
ALTER SEQUENCE SA.BASE36TBN NOORDER;
ALTER SEQUENCE SA.BASE36TCL NOORDER;
ALTER SEQUENCE SA.BASE36ACL NOORDER;
ALTER SEQUENCE SA.BASE36ACN NOORDER;
ALTER SEQUENCE SA.BASE36ACT NOORDER;
ALTER SEQUENCE SA.SEQ_BUS_SITE_ROLE NOORDER;

Database Analysis
Recommendation: Increase cache size for nominated
sequences
Increase Sequence Cache Size

You should use an increased CACHE clause and recreate the listed sequences.
One of the reasons Oracle sequence number generators are efficient is that the
numbers are cached in the Oracle shared memory area (the SGA). By default, 20
sequence numbers are cached. Once the 20 numbers are allocated, Oracle must
retrieve another 20 from the database. When sequence numbers are allocated at a
high rate, you can improve the performance by increasing the cache size when you
create the sequence. Database Analysis records the rate at which sequence numbers
are allocated and recommends an increase in the cache rate when the cache will be
exhausted in a few minutes.
Database Analysis has determined that the following sequences have a less than
optimal cache size:
Current Optimal
Sequence Name Probability
cache_size cache size
SA.SEQ_TIME_BOMB 100 185 92.4
SA.SEQ_ACT_ENTRY 100 185 92.4
Increasing the cache size does not consume memory in the SGA. However, it does
increase the number of sequence numbers that can be lost if the cache is flushed at
system shutdown.
See also:
Exploiting Oracle Sequence Generators, Re:Quest, Volume 2, Issue 2
Altering Sequences
SQL script used to alter nominated sequences storage definitions.
You can use the following SQL script to alter the nominated sequences storage
definitions. You should run this script while connected as a user with the ALTER ANY
SEQUENCE privilege.
ALTER SEQUENCE SA.SEQ_TIME_BOMB CACHE 185;
ALTER SEQUENCE SA.SEQ_ACT_ENTRY CACHE 185;

Database Analysis
Recommendation: Tune resource-intensive SQL
Optimize SQL Statements

Database Analysis has identified SQL statements that consume a significant amount of
resources. This can be improved by tuning . You should examine these statements to determine
whether their resource consumption can be reduced (optionally you can use Quest Central SQL
Tuning from the Performance Management tools).
Note: This recommendation was based on the current data held within your Oracle instance and not from
data collected by the snapshot processes. It might not, therefore, be totally applicable to the analysis
period you specified.
The following SQL statements consumed a significant portion of identified disk I/O, required a large
number of I/Os to retrieve rows from the database, or both. To display the Execution Plan for a specific
SQL statement, click the icon.
Buffer Buffer Execution

SQL statement User Name
Gets Gets/Exec Plan
SELECT COUNT(*) FROM
DBA_USERS WHERE ( HP_DBSPI 50022 50022
DEFAULT_...
select tta.objid from
table_demand_dtl tta, ZAPAJOS 558117 39866
tab...
select count(*) into :b0
from DBA_OBJECTS HP_DBSPI 268102 38300
where...
DELETE FROM
QUEST_IX_RUNS WHERE QUEST_USER 36485 36485
RUN_ID = :b1
select employee from
table_empl_user where ( MCLATHO 5304254 35599
...
SELECT /*+ */
"A1"."OCI_RMA_NU... APPS 70835 35418
FROM "SA"."CC...
SELECT COUNT(*) FROM
DBA_SEGMENTS WHERE HP_DBSPI 34728 34728
TABLESP...
select elm_objid from
table_wipelm_dtl where TINOGUA 100685 33562
...
Tuning SQL Statements

The easiest way to tune SQL statements is to use Quest SQL Tuning, one of the Quest Central
Performance Management tools.
About SQL Tuning

Quest SQL Tuning is a full-featured environment for tuning SQL. Using SQL Tuning, you can perform the
following:
• Identify SQL that requires tuning.

• View SQL execution plans in traditional, graphical, or natural language mode.
• Try changes to hints or SQL structure and immediately see the results.
• Generate optimizer statistics or rebuild indexes.
• Generate expert advice that suggests ways in which you can tune the SQL statement. This
advice can include suggestions on indexing, hints, and rewording the SQL.

Database Analysis
Recommendation: Reduce Latch free latches
Reduce Latch Free Waits

Database Analysis has identified the presence of latch contention in your
database.
Database Analysis has examined your database and has determined that latch waits
account for 26% of your overall non-idle waits. Database Analysis has calculated a
95% probability that this is a significant amount of latch contention.
The total time consumed by latch waits directly corresponds to the time spent by latch
requests not granted to a requesting process. When the latch is not available, the
requesting process sleeps and then wakes later to try again to obtain the latch. The
following table lists the latches, the number gets, misses and sleeps for your
database:
Latch Gets Misses % Sleeps

library cache 9145932 .651 16157
shared pool 1614101 .741 4881
row cache objects 3316668 .342 486
cache buffers chains 56617445 .07 56
system commit number 3053608 .102 5
enqueue hash chains 606705 .019 3
redo copy 22 95.455 2
global tx hash mapping 146600 .009 1
session idle bit 2378280 .005 1
messages 165239 .352 0
redo allocation 551278 .11 0
enqueues 775969 .052 0
latch wait list 40629 .052 0
cache buffers lru chain 73049 .047 0
undo global data 344788 .019 0
dml lock allocation 138113 .007 0
session allocation 105554 .007 0
list of block allocation 66242 .005 0
transaction allocation 125524 .005 0
archive control 2 0 0
global transaction 1832533 0 0
global tx free list 15558 0 0
loader state object freelist 130 0 0
multiblock read objects 960 0 0
sequence cache 69855 0 0
user lock 1894 0 0
sort extent pool 347 0 0
session switching 60 0 0
process allocation 425 0 0
modify parameter values 500 0 0
library cache load lock 4178 0 0
ktm global data 12 0 0
cache buffer handles 390 0 0
More About Latch Free Waits
Latches are usually held for a very brief interval. In a healthy database, there should
be little or no contention for latches. Busy databases however, often suffer
considerably from latch contention.
When a process requires a latch and cannot obtain it on the first attempt, a latch miss
will result. The session will repeatedly attempt to obtain the latch until the value of the
configuration parameter spin_count is reached. This technique is known as acquiring
a spin lock. When the session still cannot obtain the latch, then the session will
relinquish the CPU and a latch sleep will result. A latch sleep will be recorded as a
latch free wait. When the session "wakes," it will repeat the attempt to obtain the latch.
Reducing Latch Free Waits

To reduce overall latch free waits, identify and correct the latches with the
highest "sleep" rates.
Latch free waits occur when a session "sleeps" on a latch. Therefore, the most
effective way to reduce latch free waits is to identify and correct the latches that
caused the most sleeps. A table showing the breakdown of sleeps by latch type is
located on the Reduce latch free waits page.
Database Analysis offers advice about specific latches when it is possible.
Types of Latches
The latches which are used most heavily and which suffer the greatest contention are
the latches associated with the following three major areas of the SGA:
• Buffer cache latches
• Library cache latches
• Redo buffer latches
Buffer Cache Latches
Two main latches protect data blocks in the buffer cache. The first is the cache buffer
LRU chain latch, which must be obtained to introduce a new block into the buffer
cache and to write a buffer back to disk. A cache buffer chains latch is acquired
whenever a block in the buffer cache is accessed (pinned).
When contention exists for buffer cache latches, it usually indicates a database which
has very high I/O rates. It is possible to reduce contention for the cache buffer LRU
chain latch by increasing the size of the buffer cache. This reduces the rate at which
new blocks are introduced into the buffer cache.
Reducing contention for the cache buffer chains latch usually requires you to reduce
the logical I/O rates by tuning and minimizing the I/O requirements of application SQL.
You can create additional cache buffer LRU chain latches by adjusting the
DB_BLOCK_LRU_LATCHES parameter. Contention for the cache buffer LRU chain
and cache buffer chain latches can occur when a database sustains very high physical
or logical I/O rates.
Library Cache Latches
The library cache latches protect the cached SQL statements and object definitions
located in the library cache of the shared pool.
The library cache latch must be obtained before a new statement can be added to the
library cache. During a parse request, Oracle searches the library cache for a
matching statement. When it is not found, Oracle will parse the SQL statement,
acquire the library cache latch, and insert the new SQL. Contention for the library
cache latch can occur when an application generates numerous unique, unsharable
SQL (usually when literals have been used instead of bind variables). When the library
cache latch bottlenecks, try to improve the use of bind variables within your
application. Misses on this latch can also indicate that your application is parsing SQL
at a high rate and may be suffering from excessive parse CPU overhead.
The library cache pin latch must be obtained when a statement in the library cache is
re-executed. Misses on this latch occur when SQL is executed at very high rates.
There is little you can do to reduce the load on this latch, although using private rather
than public synonyms (or even direct object references such as OWNER.TABLE) can
help.
Redo Buffer Latches
Two latches control access to the redo buffer. The first, the redo allocation latch, must
be acquired before space can be allocated within the buffer. The second, the redo
copy latch, is then acquired to copy redo entries into the buffer.
Prior to Oracle version 8i, you can adjust the LOG_SIMULTANEOUS_COPIES

parameter to control the number of redo allocation latches. In Oracle version 8i, the
LOG_SIMULTANEOUS_COPIES parameter is automatically set to twice the number
of CPUs on the system. Do not modify this parameter in Oracle version 8i.
You may observe high contention for the redo copy latch. This is an artifact of the way
in which Oracle obtains this latch. Oracle requests each of the redo copy latches in
turn without waiting for the latch. Only when all of the latches are busy will Oracle spin
on the latch or enter a latch free wait state. The misses on each of the latches is still
recorded though, resulting in the appearance of a high miss rate for this latch.
Shared Pool Latch
The shared pool latch is used when sessions allocate or unallocate space within the
shared pool. Heavy contention for this latch indicates that the shared pool is too large.
Large shared pools can generate excessively long freelists. The longer the freelists,
the longer it takes to be scanned and the longer the latch is held.
Spin Count and Latch Sleeps
When a session "sleeps" while trying to obtain a latch, the database response time will
be significantly degraded. You can decrease the probability of session sleeps by
increasing the value of the _LATCH_SPIN_COUNT or SPIN_COUNT configuration
parameter. This parameter controls the number of attempts the session will make to
obtain the latch before sleeping. "Spinning" on the latch consumes CPU. Increasing
the value of this parameter can increase the overall CPU utilization of your system.
When your system is near 100% CPU utilization and you are using a throughput
application rather than a response time application, you can consider decreasing the
SPIN_COUNT value to conserve CPU.
Adjusting the SPIN_COUNT parameter value is a trial and error process. In general,
only increase the value when there are enough free CPU resources available on your
system. Decrease the value only when there is no spare CPU capacity.

Database Analysis
Graph: Activity Summary
Activity Summary
This report provides an overview of the instance activity during the analysis period. Increases in load
can result in increased contention for resources and degraded response times or throughput.
User activity
Figure 1 shows the number of sessions connected to the database during the analysis period.
"Active" users are those that are currently executing a database call (probably a SQL request). High
values for "Active" may indicate a period of contention or extreme load.
Figure 1 User activity during the analysis period
Event Wait times

In a perfect Oracle implementation, the server process would perform tasks using resources without
experiencing any delays. However, Oracle sessions usually must wait for resources to become
available. The following includes some of the resources that can cause waits:
• wait for an IO to complete
• wait for a lock to be released
• wait for space in shared memory.
Figure 2 shows a high-level categorization of the time spent waiting in various wait states. These are
examined in more detail in the 'Wait activity' report.
Figure 2 Summary of Wait times during the analysis period

Physical I/O
Physical I/O often accounts for a large portion of database activity and can often cause a bottleneck.
Figure 3 shows I/O rates to the Oracle datafiles and to the Oracle redo logs. High I/O rates are
associated with periods of excessive load. An excessive load can be caused by a single batch report
or an ad-hoc SQL statement.
Figure 3 Physical I/O during the analysis period
Logical I/O
Logical I/O occurs whenever a connected session wants to access information in the database. A
Logical I/O will only become a physical I/O when the necessary information is not present in the buffer
cache (an area of shared memory in the SGA). Three types of logical I/O can occur within the buffer
cache:
• A consistent read which requires that the data block be consistent at a given point in time.
This sort of I/O usually involves queries.
• A current read accesses of the most recent version of a block. This sort of I/O is typical of
DML (Update, Delete, Insert) operations.
• Block changes that occur when a session modifies a block in shared memory.
Figure 4 Logical I/O during the analysis period
Call rates
When a client communicates with an Oracle instance, the calls can be categorized as follows:
• request to prepare a SQL statement for execution (parse call)

• request to execute a SQL statement
• request to commit a transaction
• request to rollback a transaction.
Call rates give a good overall indication of the amount of work being done. High parse rates (relative
to executions) or high rollback rates (relative to commits) can indicate inefficiencies in the application.
Figure 5 Call rates during the analysis period

Miss Rates
Miss rates reflect the percentage of time in which a session waited for a resource. The following
miss rates are shown in Figure 6:
• Buffer cache - A miss occurs when a user issues a logical read request and the requested
block is not already in memory.
• Latch - A miss occurs when a user attempts to acquire a latch and none are available.
• Library cache - A miss occurs when the session cannot find a SQL statement in the shared
pool (library cache).
Figure 6 Miss rates during the analysis period

Database Analysis
Graph: File I/O
Datafile I/O
Physical reads by datafile
Figure 1 shows the physical read rates for the most active datafiles. High read rates can indicate
heavy database activity. However, high read rates can also be caused by poorly tuned SQL queries
that access the datafiles. Datafiles with High I/O rates should generally be spread across multiple disk
devices. You can accomplish this by either using RAID or by alternating datafiles across multiple disk
devices.
Figure 1 - Physical reads by datafile

Physical reads by tablespace
Figure 2 shows the physical read rate for each tablespace.
Figure 2 - Physical reads by tablespace

Physical writes by datafile
Figure 3 shows the physical write rate for each datafile.
Figure 3 - Physical writes by datafile

Physical writes by tablespace
Figure 4 shows the physical write rate for each tablespace.
Figure 4 - Physical writes by tablespace

Average IO time by datafile
Figure 5 shows the average I/O time (in milliseconds) of read activities for each datafile. Datafiles
displaying high average I/O times could be experiencing contention or could be located on "slow"
disks.
Figure 5 - Average IO time by datafile

Database Analysis
Graph: Latch Summary
Latch Activity
Introduction
Latches are Oracle internal locking mechanisms that prevent multiple sessions from simultaneously
updating the same item within Oracle shared memory (SGA). When a session requests a latch that is
held by another session, a latch free wait occurs.
The presence of latch free waits of any significant magnitude may indicate a bottleneck within the
SGA. The degree to which database performance is impacted depends on the latch. The following
sections describe the latches and how they can impact performance.
Latch Efficiency Indicators

The following metrics can indicate the presence of a latch bottleneck:
• Latch miss rate - the percentage of latch attempts that did not succeed on the first attempt.
• Latch sleep rate - the percentage of latch attempts that did not succeed and that eventually
required the session to "go to sleep" waiting for the latch to become available.
• Latch wait pct - the percentage of time that all sessions spent waiting for latches to become
available, as a percentage of the total time spent waiting for non-idle events.
Figure 1 shows the values for these three indicators across the analysis period:
Figure 1 - Latch Efficiency Indicators

Breakdown of Latch Waits
Table 1 shows the breakdown of latch waits and sleeps by latch type. Latches are discussed in detail
in the following sections.
Table 1 - Breakdown of Latch Waits by Type
Latch Gets Misses Sleeps Miss Rate Sleep

library cache 9145932 59566 16157 .65 .18
shared pool 1614101 11962 4881 .74 .3
row cache objects 3316668 11333 486 .34 .01
cache buffers chains 56617445 39831 56 .07 0
system commit number 3053608 3117 5 .1 0
enqueue hash chains 606705 116 3 .02 0
redo copy 22 21 2 95.45 9.09
global tx hash mapping 146600 13 1 .01 0
session idle bit 2378280 120 1 .01 0
archive control 2 0 0 0 0
cache buffers lru chain 73049 34 0 .05 0
cache buffers lru chain 73049 34 0 .05 0
global transaction 1832533 0 0 0 0
library cache load lock 4178 0 0 0 0
loader state object
130 0 0 0 0
freelist
list of block allocation 66242 3 0 0 0
sort extent pool 347 0 0 0 0
session switching 60 0 0 0 0
session allocation 105554 7 0 .01 0
sequence cache 69855 0 0 0 0
redo allocation 551278 605 0 .11 0
process allocation 425 0 0 0 0
multiblock read objects 960 0 0 0 0
modify parameter
500 0 0 0 0
values
user lock 1894 0 0 0 0
undo global data 344788 64 0 .02 0
transaction allocation 125524 6 0 0 0
messages 165239 582 0 .35 0
latch wait list 40629 21 0 .05 0
ktm global data 12 0 0 0 0
global tx free list 15558 0 0 0 0
enqueues 775969 400 0 .05 0
dml lock allocation 138113 9 0 .01 0
cache buffer handles 390 0 0 0 0
More about Latches

A latch is similar to a lock, but instead of preventing two sessions from concurrently changing the
same row, a latch prevents two sessions from concurrently altering the same area in shared memory.
Latches are usually held for a very brief interval, and in a healthy database there should be little or no
contention for latches. Unfortunately, very busy databases often suffer considerably from latch
contention.
When a process requires a latch and cannot obtain it on the first attempt, a latch miss will result. The
session will repeatedly attempt to obtain the latch until it reaches a number of attempts specified in
the SPIN_COUNT parameter. This technique is known as acquiring a spin lock. When the session
still cannot obtain the latch, then the session will relinquish the CPU and a latch sleep will result. A
latch sleep will be recorded as a latch free wait. When the session wakes, it will again attempt to
obtain the latch.
The latches that contribute to a high proportion of misses or sleeps deserve attention. The latches
which are used most heavily and which therefore typically suffer the most contention are the latches
associated with the three major areas of the SGA. These are the buffer cache latches, the library
cache latches, and the redo buffer latches.
Buffer Cache Latches

Two main latches protect data blocks in the buffer cache.
• The cache buffer LRU chain latch must be obtained before introducing a new block into the
buffer cache and also when writing a buffer back to disk.
• The cache buffer chains latch is acquired whenever a block in the buffer cache is accessed
(pinned).
Contention for these latches usually indicates that a database has very high I/O rates. It may be
possible to reduce contention for the cache buffer LRU chain latch by increasing the size of the buffer
cache. This will reduce the rate at which new blocks are introduced into the buffer cache.
Reducing contention for the cache buffer chains latch usually requires the logical I/O rates to be
reduced. This is accomplished by tuning and minimizing the I/O requirements of application SQL.
You can create additional cache buffer LRU chain latches by adjusting the configuration parameter
DB_BLOCK_LRU_LATCHES.
Library Cache Latches

The library cache latches protect the cached SQL statements and objects definitions held in the
shared pool library cache.
The library cache latch must be obtained before a new statement can be added to the library cache.
During a parse request, Oracle searches the library cache for a matching statement. When one is not
found, Oracle will parse the SQL statement, acquire the library cache latch, and insert the new SQL.
Contention for the library cache latch can occur when an application generates very high quantities of
unique, unsharable SQL. This usually occurs when literals have been used instead of bind variables.
When the library cache latch is a bottleneck, try to improve the use of bind variables within your
application. Misses on this latch may also be a sign that your application is parsing SQL at a high rate
causing excessive parse CPU overhead.
The library cache pin latch must be obtained when a statement in the library cache is re-executed.
Misses on this latch occur when there are very high rates of SQL execution. There is little you can do
to reduce the load on this latch, although using private rather than public synonyms (or even direct
object references such as OWNER.TABLE) can help.
Contention for library cache and library cache pin latches can occur when there is heavy parsing or
SQL execution rates. Misses on the library cache latch is usually a sign of excessive re-parsing of
non-sharable SQL.
Redo Buffer Latches

Two latches control access to the redo buffer.
• The redo allocation latch must be acquired before space within the buffer can be allocated.
• The redo copy latch must be acquired before redo entries can be copied into the buffer.
Prior to Oracle version 8i, you can adjust the LOG_SIMULTANEOUS_COPIES parameter to control
the number of redo allocation latches. In Oracle version 8i, the LOG_SIMULTANEOUS_COPIES
parameter defaults to twice the number of CPUs on the system. This parameter should never be
changed.
You may observe seemingly high contention for the redo copy latch but this is really an artifact of the
way in which Oracle obtains this latch. Oracle attempts to obtain each of the redo copy latches in turn
without waiting for the latch. Only when all of the latches are busy will Oracle spin on the latch or
enter a latch free wait state. However, because a miss on each of the latches is still recorded, it is
common to see a high miss rate on this particular latch. High miss rates on the redo copy latch are
normal and do not usually indicate serious latch contention.
Shared Pool Latch

The shared pool latch is required when sessions need to allocate or deallocate space within the
shared pool. Heavy contention for this latch can actually suggest that the shared pool is too big,
because large shared pools may generate excessively long freelists that are scanned while holding
the latch.
Spin Count and Latch Sleeps

When a session "sleeps" because it cannot obtain a latch, response time is significantly degraded.
You can decrease the probability of the session sleeping by increasing the value of the
_LATCH_SPIN_COUNT or SPIN_COUNT parameters. These parameters control the number of
attempts the session will make to obtain the latch before sleeping. "Spinning" on the latch consumes
CPU, so when you increase this parameter, you may see an increase in your system's overall CPU
utilization. When your computer is near 100% CPU utilization and you are using a throughput
application rather than a response time application, you can consider decreasing the SPIN_COUNT
parameter value to conserve CPU.
Adjusting the SPIN_COUNT parameter value is a trial and error process. In general, only increase the
SPIN_COUNT parameter value when there are enough free CPU resources available on your
system. Decrease the SPIN_COUNT parameter value only when there is no spare CPU capacity.

Database Analysis
Graph: Misc reports
Miscellaneous Performance Indicators

Parallel Query Activity
Figure 1 shows the number of idle and busy parallel query servers detected during the analysis
period. When all of the parallel query slaves are busy, statements requesting parallel execution may
run serially or at a reduced level of parallelism. Make sure that there is an appropriate number of
parallel servers to service the parallel processing demand. However, configuring too many parallel
servers will likely overload your system. Set the PARALLEL_AUTOMATIC_TUNING parameter to
TRUE, this will cause Oracle to select a suitable value for the number of servers.
Figure 1. Parallel Server Activity
MTS and Dispatcher Performance

In a Multi-threaded server configuration (MTS), Oracle sessions form a permanent association with a
dispatcher process. This dispatcher is responsible for allocating SQL requests to the appropriate
shared server. When the dispatcher or all servers are busy, the session will need to wait until the
process becomes available. When the Dispatcher or Shared servers are showing a high busy rate, it
is possible that you may need to add more dispatcher or shared server processes.
Note: Short term or sporadic high busy rates can severely impact response time during peak activity,
even when the average busy rate seems acceptable.
Figure 2 shows the percentage of dispatcher and server requests which had to wait. This rate should
generally be kept close to zero.
Figure 2. MTS/Dispatcher Wait Rates

Rollback Segment Activity
Figure 3 shows the rates of rollback segment wraps, extends, and shrinks. The following is a
description of these segment transactions:
• Wrap - occurs when a rollback segment has used all of the available extents and must "wrap"
to re-use the first extent. High rates of wraps can indicate that the rollback segments are too
small. Although wraps in themselves do not consume resources, they do limit the maximum
possible duration for long running queries.
• Extend - occurs when a transaction needs more space than is available in the rollback
segment. This requires more extents to be added. High rates of extends may indicate that the
setting for OPTIMAL is too low.
• Shrink - occurs when a transaction moves into an unused extent and detects that the
rollback segment is larger than the OPTIMAL setting. The rollback segment will then "shrink"
back to the size specified by OPTIMAL. High rates of shrinks may indicate that the setting for
OPTIMAL is too low.
• Wait - occurs when a session unsuccessfully attempts to access a particular rollback
segment header. High wait rates indicate a need to create additional rollback segments.
Database Analysis
Graph: Wait events
Wait Activity
Summary of Wait Events
When an Oracle session must wait for a resource to become available, it records the type and
duration of the wait to the Oracle dynamic performance tables V$SYSTEM_EVENT and other related
views. Examination of these wait conditions is one of the most important performance diagnostic
measures because many performance tuning opportunities relate to reducing the number of
unnecessary waits or to reducing the average duration of necessary waits.
Figure 1 summarizes event waits during the analysis period. Some waits, for example, those that
occur during idle periods, are eliminated. Other waits are aggregated.
Figure 1 Summary of Wait Conditions
Wait Details
Table 1. shows a more detailed breakdown of wait events during the analysis period. This breakdown
aggregates many of the categories that are contained in V$SYSTEM_EVENT and excludes waits that
are regarded as "idle" waits.
Table 1. Detailed Breakdown of Wait Times
Event Waits/Sec Waits Ms Sec Avg Wait

single block 27.01 122241 207.87 940.9 7.7
Latch 4.77 21592 108.45 490.75 22.73
SQL*NET 313.2 1417658 33.61 152.13 .11
Lock .01 42 24.64 111.49 2654.52
Log write 8.52 38568 20.8 94.16 2.44
Log sync 7.42 33567 19.22 87 2.59
db file write 1.14 5153 1.71 7.74 1.5
multi-block .05 235 .47 2.14 9.11
Control File IO .44 1979 .24 1.1 .56
Log switch 0 5 .16 .71 142
buffer busy .01 57 .01 .05 .88
Archiver 0 0 0 0 0
write complete 0 1 0 0 0
log buffer space 0 0 0 0 0
free buffer 0 0 0 0 0
direct path 0 0 0 0 0
direct path read 0 0 0 0 0
db file parallel 0 0 0 0 0
Other .01 33 0 0 0
Log switch 0 0 0 0 0
Log switch 0 0 0 0 0
Log read 0 4 0 0 0
External File IO 0 0 0 0 0
External File IO 0 0 0 0 0
File IO 0 0 0 0 0
Cluster co-
0 0 0 0 0
ordination
Buffer Busy Breakdown

Figure 2. shows the breakdown of buffer busy waits by buffer type. Most buffer busy waits are on one
of the following:
• Data block - This is generally caused by concurrent inserts in a table with insufficient
freelists. Insufficient freelists do not cause freelist buffer busy waits, and unless you
configured multiple freelists_groups (recommended only for Oracle Parallel Server) freelist
buffer busy waits do not occur.
• Undo header or block - This generally occurs when you need to configure more rollback
segments
Figure 2. Buffer Busy Wait Breakdown
Understanding Wait Events

Although Database Analysis generally makes detailed recommendations when unexpected wait types
of average durations are encountered, use the information contained in this section to interpret the
information that is contained in Table 1. and Figure 1.
Db File Waits
Events such as multi-block read, single-block read, direct path read/write, and db file write all occur
when an Oracle session issues an I/O request against an Oracle datafile.
Database file writes are performed only by the database writer. Therefore, "db file" write waits are not
experienced by user sessions. User sessions do, however, read data from database files directly and
will almost always wait for datafile read events.
Unless your entire database is cached in the SGA, waiting for database file I/O is inevitable and the
presence of "db file" waits does not indicate a problem with database performance. In most
databases, "db file" waits account for approximately 80 to 90% of all nonidle wait times.
The average wait times for db file events is an indication of the speed and efficiency of your disk I/O
subsystem. High speed disks should allow for average waits of between 1 and 2 hundredths of a
second. Higher values can indicate that you should review your disk configuration.
Log Sync/Write waits

Oracle sessions must wait for db file I/O as well as for log file I/O. Log file I/O waits occur when a
COMMIT statement causes a write to the redo log. The session issuing the COMMIT will wait on the
log file sync event. When the LogWriter process issues the I/O request, it will wait on the log file
parallel write event.
Both of these wait events are inevitable and account for between10 and 20% of total nonidle wait
times in a database. The average wait time for a log file parallel write is an important measure. It
indicates how quickly the log writer process can flush the redo buffer and is a good indicator of the
efficiency of the redo log device. Values under 1 hundredths of a second are good, and values of up
to 5 hundredths of a second are not unusual. Values above this range may indicate issues with the
redo log device.
Log File Space/Switch Waits

These events occur when a redo log entry cannot be made when there is no free space in the redo
log buffer or when a redo log cannot be written to because it is in the process of being switched.
These events should not affect your database. These waits are defined by the following events:
• Log Buffer Space. Waiting for free space in the redo log buffer. You can reduce this wait by
increasing the size of the log buffer (LOG_BUFFER parameter) or by optimizing the
performance of the logwriter.
• Log File Switch (checkpoint incomplete). The next redo log cannot be used because a
checkpoint that commenced when the log was last switched has not completed.
• Log File Switch (archiving needed). A redo log cannot be used because an archive
operation that commenced when it was last switched has not completed.
These waits suggest that either your redo logs are on slow devices, your log_buffer setting is too low,
or you have too few or too small redo logs. When you continue to see the "log file switch (archiving
needed)" event, you should alternate your redo logs across multiple devices to reduce any issues
between the LogWriter and Archiver processes.
Buffer Busy Waits
This event occurs when a session cannot access a block in the SGA because the block is in use by
another session. The two most common causes of this event are insufficient free lists for a table and
too few rollback segments.
You can use the V$WAITSTAT table to distinguish between these two causes. If the majority of the
waits are for "data block" or "free list" events, it is likely that you need to create multiple FREELISTS
(using the FREELIST clause of the CREATE TABLE statement) for tables that are subject to very
heavy concurrent inserts. When the majority of the waits are for either "undo header" or "undo block"
events, you may need to create additional rollback segments.
Free Buffer Waits/Write Complete Waits

Both of these events occur when a session attempts, but is unable to modify or insert a block in the
SGA.
"Write complete waits" occur when a session attempts to modify a block that is currently being written
to disk by the database writer process. This occurs occasionally, particularly during checkpoints.
However, when it is causing a large number of waits, it may indicate inefficiencies in the database
writer.
"Free buffer waits" occur when a session attempts to read a data block from a database file on disk to
the buffer cache. When there are no unmodified (clean) blocks in the buffer cache, the session must
wait for the Database Writer process to write modified (dirty) blocks to disk so that free buffers can be
available. However, the database writer is constantly writing dirty buffers to disk, therefore, this event
should occur only rarely.
When either of these events make up the majority of total waits, you may need to improve the
performance of the Database Writer process (DBWR). You can perform the following to complete this
task:
• Use asynchronous I/O or multiple database writers (DB_WRITERS parameter).

Asynchronous I/O is enabled by default under NT. In UNIX you must set ASYNC_IO=TRUE
in your server parameter file and you may have to enable asynchronous I/O at the operating
system level. It is recommended that you use asynchronous or list I/O rather than multiple
database writers when possible.
• Stripe your datafiles across multiple disk devices. The number of devices across which the
datafiles are housed and the evenness of the spread determines the ultimate I/O limit of your
system. Use operating system striping if possible (RAID-0). However, when this is not
available, alternate datafiles across multiple disk devices.
• Avoid RAID-5. RAID-5 can be attractive because it spreads I/O across multiple devices and
enables fault tolerance more economically than mirroring (RAID-1). However, RAID-5 can
decrease the write capability of your disks by half since each local write requires at least two
physical writes (additional I/Os are required to read and write the parity block). Many RAID-5
vendors provide large nonvolatile disk caches in an attempt to avoid this write penalty. Free
buffer or write complete waits may be a sign that these efforts are unsuccessful.
• Consider raw devices/partitions. The use of raw devices (unformatted disk devices without an
overlying file system) is somewhat controversial and may not be available for all applications.
However, the database writer process generally benefits from raw devices.
Lock ("Enqueue") Waits

Enqueue waits occur when a session waits to obtain a lock. In most cases, this occurs because of a
lock on a table or row that the session wants to lock or modify. At times, the lock involved may be an
ORACLE internal lock (for instance, the Space Transaction enqueue). When the database is well
tuned and the application design is sound, enqueue waits should be negligible. The following are
common causes of excessive enqueue waits:
• Contention for a specific row in the database. The application design may require that many
processes update or lock the same row in the database. One common example of this is
when primary keys are generated using a sequence table.
• Table locks caused by unindexed foreign keys. When an unindexed foreign key is updated,
the parent table is locked until the transaction is complete.
• "Old-style" temporary tablespaces. When a temporary tablespace has not been created with
the TEMPORARY clause (introduced in ORACLE 7.3), sessions may contend for the "space
transaction" lock.
• The space reserved for transactions in a data block is too small. By default, only one
transaction slot for tables and two for indexes is allocated when the table or index is created.
The number of transaction slots is determined by the INITRANS clause in the CREATE
TABLE or INDEX statement. When additional transaction slots are required, and there is free
space in the block, they are created. However, when all transaction slots are in use and there
is no free space in the block, a session that wants to lock a row in the block encounters an
enqueue wait, even when the row is not actually locked by another process. This can occur
when both PCTFREE and INITRANS are set too low.
Latch Free Waits

Latches are Oracle internal locking mechanisms that prevent multiple sessions from updating the
same item at the same time in Oracle shared memory (SGA). When a session must acquire a latch
that is held by another session, a latch free wait may occur.
A large number of latch free waits may indicate a bottleneck in the SGA. Since latches are such a
significant performance diagnostic, a complete rule pack and report page is dedicated to latch
analysis (for further information, see the "reduce latch contention" goal and the "Latch analysis"
report).
SQL*Net Waits
The Oracle server often records that it is waiting for the completion of a SQL*NET operation. These
waits all begin with "SQL*NET". It can be difficult to isolate the waits that are the result of network
overhead from those that signify that the client process is busy performing other tasks. You should
ignore the "SQL*Net message from client" event, which occurs when the server process is idle and is
waiting for further instructions from the client process. When other SQL*NET waits occur frequently,
you should investigate your network overhead. However, these waits can occur when bottlenecks are
present in the client program or in a remote server (as indicated by "db link" waits).

Healthcheck Detect Diagnose Resolve

Uploaded by

Copyright:

Available Formats

You might also like

Healthcheck Detect Diagnose Resolve

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Healthcheck Detect Diagnose Resolve

Uploaded by

Copyright:

Available Formats

Oracle Health Check

Section Two: Database Tuning Methodology: Detect, Diagnose, Resolve

Section Three: Sample Report—Excerpt of Health Check done in September 2002

Why Application Management?

Quest for Application Confidence

Quest Software’s Performance Tuning Methodology:

Foglight Notice Board Within Quest Central

Intuitive User Interface Predefine actions

Data Repository for Point-in-time Analysis

Maintaining Service Level Agreements Schedule reports

Enhancing the End User Experience

Foglight offers an array of specialized cartridges focused on your e-business applications,

• Foglight Cartridge for mySAP.com™

Quest Central for Oracle:

Performance Diagnostics Components:

• Provides an intuitive, architecturally correct GUI of process flows within an Oracle

Database Administration Component

• Quickly create, drop or alter database objects

Another important aspect of database administration is managing security and permissions.

SQL Tuning Component, previously SQLab Vision

• Captures all SQL statements and performance metrics non-intrusively at sub-second

Space Management Component, previously Space Manager

Quest Central’s Space Management module simplifies time-consuming, error-prone methods

Space management provides a full range of fast reorganization methods—including

This historical information is used to identify objects in need of reorganization and to

Enclosed, you will find the following:

About the Communications Company Analyzed:

Customer: Communications Provider

Baseline Discovery Interview

One-day Consulting Engagement for Installation and Training

Five-day Automated Data Collection

Receive Report from Quest Sales Representative

Keep the Report

Get Started Today!

Performed at: Communications Provider (“Company”)

Comments and Findings

Recommendations Benefit Effort

Tune resource-intensive SQL High High

Graph 1 shows that from 3:30PM to

Graph 2 shows that from 3:30PM to 6:15PM on

Graph 3 shows that from 3:30PM to

Recommendation Confidence Effort Benefit

Goal/Recommendation Confidence Effort Benefit

This report was generated by Database Analysis.

• specific rows in the database

The following enqueue waits have been detected:

This report was generated by Database Analysis.

Tables With Lock Waits

Lock waits have been observed for the following tables:

Table Number of observations

Lock Waits for Tables

Some of the main causes of table locking include the following:

This report was generated by Database Analysis.

Reduce Space Transactions

Space transaction locks can indicate the following:

• "Old-style" temporary tablespaces. When the tablespace nominated as the

Locally Managed Tablespaces

Reduce Space Transaction Waits