1/11/2019 Document Display

Copyright (c) 2019, Oracle. All rights reserved. Oracle Confidential.

Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite (Doc ID
1121043.1)

APPLIES TO:

Oracle E-Business Suite Performance - Version 11.5.10.2 to 12.1.3 [Release 11.5.10 to 12.1]
Information in this document applies to any platform.

PURPOSE

This document outlines the most common performance diagnostic data used by development for analyzing and resolving
performance issues in Oracle E-Business Suite from version 11.5.10 to R12.2.X and DB from version 10.2.X to 11.2.X. . It
describes how to produce, collect, and format this data.

SCOPE

The following topics are discussed:

What data to collect when diagnosing performance issues in Oracle E-Business Suite
How to collect this data correctly, and avoid common mistakes
Step-by-step instructions on using SQL Trace in Forms and Self-Service based applications, Concurrent Processing,
and SQL*Plus
Steps to obtain in-memory runtime execution plans (with row source counts/statistics) when tracing does not show
this data

DETAILS

First Pass Diagnostic Data

When Oracle Support work remotely with customers on performance issues, it is desirable to strike a balance between
obtaining sufficient information on the first pass (in order to minimize back-and-forth requests to the customer until issue
resolution), and gathering too much data (which can be tedious and significantly impact the test results).

To this end, the information here is structured not by technology stack component or version, but by how often it is
relevant to the performance issues received from the field. For example, while there are cases where only the output from
a PL/SQL profiler report will contain the information necessary to identify the issue, in practice this is found to be the case
less than 5% of the time. Therefore, this information will not be requested in the first pass.

Close to 90% of the performance issues worked on by Oracle E-Business Suite Development are related to SQL
processing. This does not necessarily mean it is necessarily an SQL bug: on many occasions, the root cause is found to lie
in such areas as incorrect statistics, init.ora parameters, DB server workload management, or Oracle RAC configuration.
Consequently, the first set of data requested is focused on obtaining all necessary information to diagnose and identify the
root cause of such issues.

At a high level, the following four items should initially be requested for any performance issue with a program or user
action:

SQL Trace (Level 8, i.e. with waits) - raw trace and TKPROF
SQLT XTRACT report for poorly-performing SQL statements in TKPROF/AWR
AWR report for the period when the problem occurred
Operating system statistics for the period when the problem occurred

In 10% or fewer of cases, the main performance issue turns out not to be related to database processing, and a second
set of diagnostic data will likely be requested. This will include additional information from application tier client side
processes such as Reports, Forms, and Java.

To avoid losing time with back-and-forth requests for diagnostic data, the above data should be reviewed to validate if it is
accurate and complete:

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 1/9
1/11/2019 Document Display

Trace file should be complete as defined below in the section "How can I tell if my trace file is complete?"
Quickly look over the produced SQLT XTRACT reports to confirm that the expensive SQL statements of the flow are
covered
Ensure the AWR report and the OS stats provided are for the period of time the performance issue occurred

SQL Trace (Level 8, i.e. with waits) - raw trace and TKPROF

This is by far the most important diagnostic in cases where the issue is related to SQL processing, which, as stated, is the
majority of cases. However, not all traces and accompanying TKPROFs contain the all the information they are hoped to
have. So it is necessary to clarify not only the methods (most of which are well known) for capturing SQL traces in Oracle
E-Business Suite, but also to define the difference between a useful (complete) trace and a poor trace; how to check
which one you have; and how to identify common problems that can prevent complete data from being captured in the
traces you produce.

A useful trace and TKPROF will provide the information necessary to identify how much time is being spent in SQL
execution, PL/SQL execution, and to some extent also how much time is being spent in client processing and network
communication between the client and server processes. It will also clearly identify which SQL statements are taking most
time, how much time is being spent on database waits and which ones. It will also give profiling data for internal execution
of SQL statements, listing the runtime execution plans with actual returned row counts and statistics (operation counts,
CPU/elapsed timing, disk IO, logical IOs, etc) for each execution plan operation or rowsource.

In short, a useful trace and TKPROF will usually provide sufficient data to pinpoint the what, where and to some extent the
why of a performance issue. For cases where the complete answer is not so identified, it highlights the areas that need to
be looked at more closely to arrive at the complete root cause analysis.

Level 8, or trace with waits, should be used by default. On a few occasions, bind variable value information are also useful.
However, as this is not common, and adds considerable overhead to many SQL statements (especially if executed row by
row, or with a high number of bind variables), it should not be gathered initially.

For step-by-step instructions on recommended method for enabling SQL trace in Oracle E-Business Suite for common
components such as Forms, Self-Service or HTML flows, Concurrent Programs, or simply from SQL*Plus, see Appendix A:
Correctly Enabling SQL trace in Oracle E-Business Suite.

Recommendations for successful SQL tracing

For Oracle Database 11g, ensure these tracing related patches have been applied to the system: Patch 7522002,
Patch 8342329, Patch 11687336, and Patch 8328200. These fixes address critical issues related to timing data
accuracy and missing rowsource counts and statistics in SQL traces.

For online flows, log out of the application after performing the action to be traced. After logging out, pick up the
raw trace file and run TKPROF. Do not turn trace off (unless tracing a Self-Service flow, which uses connection
pooled sessions).

Make sure you use the same version of TKPROF as your database. For example, running TKPROF from an 8.0.6
Oracle Home will cause the timing data to be invalid, due to differences in timing units between database versions.

Ensure there is sufficient disk space in the tracing directory, and that the trace file size limit is cleared (set the
parameter max_dump_file_size=unlimited).

It is not necessary to use the explain option in TKPROF, as the interest is only in the runtime execution plans
captured during executions and tracing, rather than in the explain plan command type output TKPROF runs later
when this option is used.

Use the sort options = fchela,exeela,prsela.

Review the TKPROF output to ensure runtime execution plans and row source statistics have been captured: these
are essential for efficient and effective performance analysis.

Trace file location and name

Traces are located in the directory defined by the init.ora parameter user_dump_dest for Oracle 10g databases, or Oracle
11g if the parameter diagnostic_dest is not set. For Oracle 11g and higher databases with parameter diagnostic_dest set,
trace files are located in (diagnostic_dest)/diag/rdbms/LOWER(dbname)/(instname)/trace. (Refer note id 396009.1 for this
parameter setup.)

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 2/9
1/11/2019 Document Display

For traces generated through Forms or Self-Service flows in Oracle E-Business Suite, trace file names will include the
applications username used to trace the session: for example, TRN12_ora_11065_JSMITH.trc. For concurrent programs
traced in Oracle E-Business Suite, the applications user name and the concurrent request_id will be included in the trace
file name: for example, TRN12_ora_11065_JSMITH_CR15491013.prf. If the profile option method was used to enable
trace, and a trace file identifier value was defined there, that value will be used.

How can I tell if my trace file is complete?

The easiest way is to look at the very last line in the TKPROF output, which will mention the total time captured in the
trace: look for the phrase "elapsed seconds in trace file". Ensure this closely correlates to the time reported by the users,
or to other log or observation timings. If it does not, one of the following three issues may be responsible:

Trace file is incomplete, or tracing has been turned off before the end of execution. Check for the first of these
cases by reviewing the end of the raw trace file for an incomplete print of the final line (file system full), or a
message stating the maximum dump file size has been reached (max_dump_file_size was not set to unlimited).
Check for the second of these cases by looking in the TKPROF output for a command that turns tracing off (such as
ALTER SESSION SET SQL_TRACE=FALSE)

Part of the execution of the program or flow occurred either before trace was enabled for the session, or after all
database activity for the traced session had completed.

The program or flow uses multiple database sessions, such as cases related to connections pools, database link
based activity, or child requests. Such a case necessitates locating and reviewing all relevant trace files

How can I tell if the runtime execution plans and rowsource counts and statistics are present?

In the TKPROF output, the heading "Row Source Operation" identifies the actual execution plan used at runtime and
provided that some rows where returned for at least one of the objects accessed, the execution plan should show at least
some non-zero row counts under the 'Rows' column (on the left of the plan). This can be seen in the example below,
which illustrates a complete runtime execution plan with proper rowsource data. In the raw trace file, this information is
written in lines that begin with the keyword STAT, followed by the cursor number to which they belong.

Rows Row Source Operation

------- ---------------------------------------------------
7674758 SORT AGGREGATE (cr=153846532 pr=225987 pw=0 time=7174772779 us)
7675089 NESTED LOOPS (cr=153846532 pr=225987 pw=0 time=6919378560 us)
7675089 SORT UNIQUE (cr=123146176 pr=162809 pw=0 time=6083080435 us)
55878643 TABLE ACCESS BY INDEX ROWID AR_TRX_COGS_GT (cr=123146176 pr=162809 pw=0 time=5577872837 us)
136390387 INDEX RANGE SCAN AR_TRX_COGS_N1 (cr=23439707 pr=37706 pw=0 time=2136416980 us)(object id..
7675089 TABLE ACCESS BY INDEX ROWID RA_CUSTOMER_TRX_LINES_ALL (cr=30700356 pr=63178 pw=0 time=539908369..
7675089 INDEX UNIQUE SCAN RA_CUSTOMER_TRX_LINES_U1 (cr=23025267 pr=2820 pw=0 time=196155435 us)(object
..

If, on the contrary, the only execution plan output seen for an expensive SQL statement is similar to what is shown in the
example below, this indicates that the trace is not complete, lacks runtime execution plans and row source statistics, and
is of little value for debugging the SQL performance issue.

Rows Execution Plan

------- ---------------------------------------------------
0 SORT AGGREGATE (cr=0 pr=0 pw=0 time=0 us)
0 NESTED LOOPS (cr=0 pr=0 pw=0 time=0 us)
0 SORT UNIQUE (cr=0 pr=0 pw=0 time=0 us)
0 TABLE ACCESS BY INDEX ROWID AR_TRX_COGS_GT (cr=0 pr=0 pw=0 time=0 us)
0 INDEX RANGE SCAN AR_TRX_COGS_N1 (cr=0 pr=0 pw=0 time=0 us)(object id..
0 TABLE ACCESS BY INDEX ROWID RA_CUSTOMER_TRX_LINES_ALL (cr=0 pr=0 pw=0 time=0 ..
0 INDEX UNIQUE SCAN RA_CUSTOMER_TRX_LINES_U1 (cr=0 pr=0 pw=0 time=0 us)(object ..

Execution plans listed under the heading "Execution Plan" correspond to the tkprof time explain plan command style plan,
and not the runtime plan. If you do not see the runtime row counts and statistics in the execution plan listed, verify the
trace has taken into account all recommendations made here, and re-test. If you are still unable to produce a TKPROF
with this data, this may be due either to connection pool scenarios, or because the program/session fails to finish, errors
out, or is killed. In any of these cases, you will need to use an alternative method, either tracing a particular SQL
execution manually from SQL*Plus, or accessing in-memory runtime statistics from a partial run of the flow. Instructions
for these alternative methods are given in Appendix B: Collecting In-Memory SQL Execution Runtime Statistics.
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 3/9
1/11/2019 Document Display

Note that in Oracle Database 11g (with the above tracing patches applied), partial runtime statistics are dumped into the
trace file always. By default, statistics for the first execution of the cursor are dumped, and with the new tracing option of
plan_stat=adaptive (PATCH 8328200), sampled runtime statistics are dumped for every minute of database time
consumed by an SQL statement; with this behavior, even incomplete runs will result in traces and TKPROFs with the
desired runtime performance data.

Recommended TKPROF options

For convenience of the reviewer, the most important option to use in TKPROF is the sort option. Most often, it is desired to
sort by elapsed time, so the key sort=fchela,exeela,prsela is preferred. On some occasions, however, supplying an
unsorted TKPROF can help Development more quickly locate the portions of code that are impacting performance.

As mentioned above, the explain option is not required, since the most important information regarding execution plans is
the runtime data, which is stored in the raw trace file and reported on automatically in TKPROF. Often, these two
execution plans will not be the same, and time should only be spent on the runtime execution plan.

Parallel Query traces

When tracing sessions or programs that execute one or more SQL statements which execute in parallel (whether parallel
query, parallel DML or parallel DDL), SQL processing is executed by the parallel slave processes, and they are also traced
automatically. It is therefore important to locate, identify and TKPROF these traces and have them ready for review, along
with the regular trace produced for the main (or parent) session or program being traced.

Parallel slave traces are located in the directory defined by the init.ora parameter background_dump_dest for Oracle 10g
databases, or 11g databases if the parameter diagnostic_dest is not set. For Oracle 11g and higher databases with the
parameter diagnostic_dest set, parallel slave traces are located in
(diagnostic_dest)/diag/rdbms/LOWER(dbname)/(instname)/trace. (Refer note id 396009.1 for this parameter setup.)

The trace file names will only include the instance name, parallel slave process number and process ID, so to locate the
correct set of files it may be necessary to match the timestamps in the files with the times the flow was executed flow. An
example file name is desa_p000_15630.trc.

SQLT XTRACT report for poorly-performing SQL statements in TKPROF/AWR

SQLTXPLAIN (or SQLT) is a tool developed and maintained by Oracle Support to help diagnose poorly-performing SQL
statements. SQLT output is often requested by Support and Development to expedite the analysis of SQL performance
issues. In a single-pass operation, it compiles information which helps determine why the current execution plan was
chosen, giving valuable insight into how SQL execution performance can be improved by targeting the root cause of the
inefficiency. The latest version of the tool should always be used. For details of SQLT, refer to My Oracle Support
Knowledge Document 215187.1. This document should also be used for troubleshooting any difficulties encountered using
the tool.

Although the tool supports three flavors of report (XPLAIN, XTRACT and XCUTE), the XTRACT report is preferred by Oracle
E-Business Suite Development because it focuses on extracting actual SQL execution runtime data as run by the
application.

When gathering the diagnostic data for a performance issue, it is advisable to browse the AWR and TKPROF data,
identifying the top (worst) performing SQL statements (by sql_id). For these statements, it is recommended to run and
provide the SQLTXTRACT report.

As noted previously, one way to identify these sql_id values is through the AWR report, specifically by looking at the SQL
statements listed in the section titled "SQL Ordered by Elapsed Time" (skip any PL/SQL calls listed here). Other methods
include using Oracle Enterprise Manager, directly querying V$SQL and looking at TKPROF (Oracle 11g has the sql_id
values for most SQL statements), or the raw trace file (on UNIX, grep for sql text and obtain the value following the
"sqlid="string in the preceding line).

AWR report for the same period

From Oracle Database 10g, instance performance data is stored and collected automatically. This feature is referred to as
Automatic Workload Repository (AWR). For details on its use, review My Oracle Support Knowledge Document 748642.1,
as well as the standard Oracle Database documentation.

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 4/9
1/11/2019 Document Display

Key AWR features include:

The report file is called $ORACLE_HOME/rdbms/admin/awrrpt.sql.

The HTML version of the report is preferred.

Default snapshot interval is 1 hour and default retention period is 7 days. If the storage space is available, it is
advisable to increase the retention period to at least 30 days.

If using RAC, and the Top Wait Events section shows significant waits on events with "gc" in their name, provide
one AWR report per instance for the same period, and an AWR Global report to aggregate this data. The latter
report will highlight statistics differences between nodes, and is run from the directory
$ORACLE_HOME/rdbms/admin as either runspawrrac.sql (Oracle 10gR2 and 11gR1) or awrgrpt.sql (Oracle 11gR2).
These report(s) should be verified as having been produced for the period when the performance issue occurred.

Operating system statistics for the same period

While AWR and trace data will give a useful initial indicator of whether IO performance is an issue, the CPU utilization and
memory related statistics included in those files can be limited depending on the database version, platform, and duration
of the performance issue. It is therefore also advisable to use OS tools such as sar to collect operating system resource
utilization data for the Oracle database.

Oracle Support's OS Watcher tool (OSW) provides a convenient way of collecting and present this data graphically. OSW is
a series of shell scripts that collect specific kinds of data, using UNIX operating system diagnostic utilities. Once the data is
collected, the OSWg utility (bundled with OSW) can be used to produce the graphs for the desired time interval. For
details of OSW, refer to Document 301137.1. For more details on running OSWg, see Document 461053.1.

Second Pass Diagnostic Data

As mentioned previously, the vast majority of performance issues that come to Oracle E-Business Suite Development have
their root cause associated with SQL execution, and the First Pass Diagnostic Data is meant to provide all the necessary
data to investigate and performance root cause analysis.

Sometimes, however, the issue lies elsewhere, in which case additional diagnostic data will be required. The information
requested will depend on what the First Pass Diagnostic Data shows, as well as the technology stack components present
in the flow or program being investigated. As such, a deterministic set of additional data cannot be made for all cases.

Bearing this in mind, information that will likely be requested at this stage may include:

PL/SQL Profiler - For concurrent requests, Forms and OAF/Self-Service flows, follow the instructions in Appendix A to
enable trace. In the screen where the trace level is chosen, an option is given for enabling and disabling the PL/SQL
profiler. See My Oracle Support Knowledge Document 808005.1 for additional details.

JVM log and Configuration files - The easiest way to provide these is to tar and compress the $INST_TOP directory.

Forms Runtime Diagnostics - For R12.0 and higher, see My Oracle Support Knowledge Document 445166.1; for
Release 11i, see My Oracle Support Knowledge Document 150168.1.

Reports Client Process Trace - Follow the instructions in Appendix A for enabling trace in Concurrent Requests. In
the screen where the trace level is chosen, an option is given for enabling "Reports Trace".

App Tier OS Stats - If a significant portion of the response time has found to have been spent in the application server,
OS statistics for the application tier should be provided. Oracle Support's OS Watcher tool (described in My Oracle Support
Knowledge Document 301137.1) is a convenient way to collect and present this data graphically.

It is also useful to obtain and report the amount of CPU time spent by the particular client process servicing a request, and
the amount of memory allocated by the request. In the case of Forms and OAF/Self-Service flows, this information can be
obtained as follows:

Forms - Via the menu option: Help -> About Oracle Applications

OAF/Self-Service - JVM process id (PID) and application server is given by the CLIENT_PROCESSID value in the
"Java System Properties" tab of the page displayed when clicking on the "About" link at the bottom of the page

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 5/9
1/11/2019 Document Display

(activated when profile FND_DIAGNOSTICS set to YES). Using this PID, you can use an OS command such as "ps"
to obtain its CPU and memory consumption.

Appendix A: Correctly Enabling SQL Trace in Oracle E-Business Suite

Steps for Forms Based Applications

1. Log in to the environment from which you want to trace

2. Set the value for profile option "Utilities:Diagnostics" to "Yes" at user-level for your application account.
3. Choose your responsibility, and navigate to the starting point of the form when you want to begin tracing.
4. Turn on Tracing by using the menu option: Home > Diagnostics > Trace > Trace with waits
5. A popup note with the trace file name and location is displayed. Note the trace filename. The location will be on the
database server.
6. Proceed with the application steps that need to be traced. Once finished tracing, exit the application.
7. Retrieve the raw trace file using the filename from Step 5
8. Format the raw trace file with TKPROF.

Steps for HTML/Self-Service Based Applications

1. Log in to the environment from which you want to trace

2. Set the value for the profile option FND:DIAGNOSTICS to "Yes" at userlevel.
3. Log into Self-Service Applications of that environment
4. Click the 'Diagnostics' icon at the top-right of the page.
5. Select 'Set Trace Level' radio button and click 'Go'.
6. Select 'Trace with waits' (recommended) and click 'Save'
7. Select 'Home' and proceed to perform your screen processing.
8. Disable trace once you are finished tracing your screens, with these steps:
Click on 'Diagnostics' icon > Set Trace Level > Disable Trace
9. Write down the 'Trace Ids' shown on the left side of the screen (note that tracefile_identifier is by default set to the
userid.
10. Log out of the application.
11. Retrieve the raw trace file using the process ids from Step 9 or the tracefile_identifier (set to userid by default)
12. Format the raw trace file with TKPROF.

Tracing Concurrent Programs

Steps to enable trace for Concurrent Requests: Request Level

1. Log in to the environment from which you want to trace.

2. Set the value for the profile option Concurrent: Allow Debugging to "Yes" at user level.
3. Choose the appropriate responsibility and concurrent program to be executed.
4. Click on the "Debug Options" button.
5. Enable tracing by selecting the "SQL Trace" Check box, and choose the trace level from the DropDown list. Confirm
your selection by clicking the "OK" button.
6. Submit the concurrent program.

[Perform Steps 1-6 for each individual concurrent request that needs to be traced.]
7. Write down the request_id of your concurrent program job.
8. Once the concurrent request completes, retrieve the raw trace file using the request_id (from Step 7) or the
tracefile_identifier (set by default to the userid).
9. Format the raw trace with TKPROF

The Debug Options button used for tracing individual requests is not available for request sets. You must set the
Initialization SQL Statement Custom profile option method (described below) in order to trace request sets. With this
method, a trace file will be produced for each program within the request set.

Steps to enable trace for Concurrent Requests - Program Level

1. Log in to the environment from which you want to trace.

2. Choose an appropriate responsibility and select the Concurrent > Program > Define screen.
3. Press [F11] to enter query mode and search for the concurrent program. Press [CTRL][F11] to execute the search.

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 6/9
1/11/2019 Document Display

4. Check the 'Enable Trace' box to turn on tracing for the concurrent program. Note that once checked, anyone who
runs this concurrent program will produce a level 8 (wait events) trace.
5. Submit and run the concurrent program.
6. Write down the request_id of your concurrent program job.
7. Go back to the Concurrent > Program > Define screen and uncheck the enable trace box for this concurrent
program.
8. Retrieve the raw trace file using the request_id (from Step 6) or the tracefile_identifier (set by default to the
userid).
9. Format the raw trace file with TKPROF.

Profile-Based Tracing

An alternative method to enable trace from any location at the application user level is to set the profile option
"Initialization SQL Statement - Custom" at user level, as described next.

1. Log in to Forms for the desired application.

2. If you are activating trace for your own account, navigate to the Profile > Personal menu option, press F11 to
query, enter "Initialization%" in the Profile Name column, then press Ctrl-F11 to search for this profile option. If
you are enabling trace for another user, select +Profile >> System ('Find System Profile Values' screen will pop
up).
3. Check 'User' and Type in the Username (in which the account for that user will be traced).
4. Enter 'Initialization%' in the Profile box and press 'Find'.
5. In the User box for "Initialization SQL Statement - Custom", enter the following statement and then press 'Save':
BEGIN
FND_CTL.FND_SESS_CTL('','','TRUE','TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER=''TESTING''
MAX_DUMP_FILE_SIZE=unlimited EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 8''');
END;

It is advisable to specify an identifier to help identify your trace. In this case, TESTING will appear at the end of the
trace file name. Quotes in the statement are all single quotes.
6. Log in to the application and page you wish to trace with a suitable username/password, and start your flow or
submit your request set. Everything will be traced.
7. Log out of the application once you completed your flow.
8. Go back to the Profile option in the Forms application, delete the Initialization SQL statement, and press 'Save'.
9. Identify and retrieve the trace file using the tracefile_identifier specified in Step 5.
10. Format the raw trace file with TKPROF.

SQL*Plus-Based Tracing

1. Log in to SQL *Plus

2. Run the following query to find out the server process id for your session:
SQL>select p.spid
from v$session s, v$process p where s.paddr = p.addr and s.audsid =
to_number(sys_context('USERENV','SESSIONID'));

3. Write down the server process ID (Server PID).

4. Turn on and configure tracing by executing the following commands in your SQL *Plus session:
SQL>alter session set tracefile_identifier='';
SQL>alter session set events '10046 trace name context forever, level 8';
SQL>alter session set statistics_level='ALL';
SQL>alter session set max_dump_file_size= unlimited;

5. Declare and populate the necessary bind variables used by the statement, for example:
SQL>variable B1 number;
SQL>variable B2 varchar2(50);
SQL>exec :B1 := 123456;
SQL>exec :B2 := 'ABC';

6. Run the SQL or PL/SQL code to be traced using the same bind variables as would be used by the application.
7. Once completed, run the following command, then exit the SQL *Plus session:
SQL>exec dbms_session.reset_package;

8. Identify and retrieve the trace file using the process_id from Step2 or tracefile_identifier set in Step 4.
9. Format the raw trace file using TKPROF.

Appendix B: Collecting In-Memory SQL Execution Runtime Statistics

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 7/9
1/11/2019 Document Display

As stated previously, runtime execution plan data with rowsource counts and statistics can be essential in helping to
identify the root cause of SQL execution-related performance issues. While following the SQL tracing instructions in this
document will result in complete trace files with this information in most cases, there are some cases where it will not.
Examples of these cases include long-running SQL statements that do not complete or exit with an error, or cursors that
are not closed even after logging out of the application. In such cases, it is common for the trace files not to contain the
critical STAT# lines regarding the top SQL statements you wish to analyze and optimize.

It is possible to obtain this information from memory: however, this data will age out some time after the statement of
interest is no longer executing (in Oracle Database 11g, some of this data is persisted into the AWR repository as part of
the ASH samples).

Producing and reporting on this in-memory runtime execution data is greatly simplified in Oracle 11g compared to 10g.

Oracle 11g

1. Identify the SQL_ID for the problem SQL statement using any of the traditional methods - Oracle Enterprise
Manager, AWR, ASH, V$ views, raw SQL trace, or TKPROF.
2. Report on the data, which is automatically gathered in Oracle 11g via:
Real time SQL Monitoring Report - DBMS_SQLTUNE.REPORT_SQL_MONITOR
Active Session History (ASH) Report (Section: Top SQL with Top Rowsource) -
$ORACLE_HOME/rdbms/admin/ashrpt.sql

HTML versions of the reports are preferred. While these reports do not contain all the diagnostic data produced in a
complete trace (query and current gets per row source are missing, for example), it does contain the majority of the data
needed to perform the root cause analysis of SQL execution related performance issues in most cases. Note the SQL
monitoring data is automatically collected in memory for SQL statements consuming over 5 seconds of CPU, as well as for
all parallel query/DML statements.

Oracle 10g

These steps will also work in Oracle 11g.

1. Identify the SQL_ID for the problem SQL statement using any of the traditional methods - Oracle Enterprise
Manager, AWR, ASH, V$ views, raw SQL trace, or TKPROF.
2. Ensure the cursor is not in the shared pool, either with the forceful approach of executing "alter system flush
shared_pool" (not recommended for production systems) or the more selective method (from 10.2.0.4 onwards) of
flushing only the cursor in question using these commands:
SQL> select address, hash_value
from v$sqlarea where sql_id = &1
SQL> exec sys.dbms_shared_pool.purge ('&address, &hash_value','c');

(Note that the package dbms_shared_pool must have been created via running by running (as SYS) the script
$ORACLE_HOME/rdbms/admin/dbmspool.sql.)
3. The session that will execute the SQL in question needs to run the following command before executing the SQL
that is to be diagnosed:
SQL> alter session set "_rowsource_execution_statistics"=true ;

Setting statistics_level=all also works, but with a higher overhead. If you are testing from SQL*Plus, you can
instead use the hint /*+ gather_plan_statistics*/. If required, these session settings can be set from the profile:
"Initialization SQL Statement: Custom", otherwise either via a debug package, or straight from SQL*Plus if testing
from there.
4. Enable SQL Trace (level 8, i.e. with waits) using the regular methods.
5. Let the SQL execute via your preferred method (UI, Concurrent Request, SQL*Plus, etc).
6. After the SQL has been running for some significant time, and after the program/session ends (by error, or user
action to ensure the session has ended), you can run the following commands to obtain the runtime statistics, as
well as past plans recorded in AWR:
SELECT * FROM
table(dbms_xplan.display_cursor('&sql_id',null,'ALLSTATS')) ;

SELECT * FROM
table(DBMS_XPLAN.DISPLAY_AWR('&sql_id',null,null,'ALL'));

The output should look similar to this:

PLAN_TABLE_OUTPUT
-----------------------------------------------------------
SQL_ID 1ah5hj92zuphp, child number 0

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 8/9
1/11/2019 Document Display
-------------------------------------
select count(*) from gl_je_lines a, gl_je_lines b

Plan hash value: 630681783

----------------------------------------------------------------------------------------------------
-------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | O/1/M |
----------------------------------------------------------------------------------------------------
-------------------------

| 1 | SORT AGGREGATE | | 1 | 1 | 0 |00:00:00.01 | 0 | | | |

| 2 | MERGE JOIN CARTESIAN| | 1 | 1719G| 373M|01:02:16.37 | 3123 | | | |

| 3 | INDEX FULL SCAN | GL_JE_LINES_U1 | 1 | 1311K| 284 |00:00:00.01 | 3 | | | |

| 4 | BUFFER SORT | | 284 | 1311K| 373M|00:24:54.78 | 3120 | 18M| 1581K| 1/0/0|

| 5 | INDEX FULL SCAN | GL_JE_LINES_U1 | 1 | 1311K| 1315K|00:00:05.26 | 3120 | | | |

----------------------------------------------------------------------------------------------------
-------------------------

In the above output (from the first command executed), column A-Rows, A-Time, Buffers and Reads (not shown on the
example) are actual runtime statistics for each execution plan rowsource, up to the point where the execution ended.

Note: Some of the features described in this document, such as Oracle 11g Real Time SQL Monitoring, may require
additional licensing.

REFERENCES

NOTE:1121043.1 - Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite
NOTE:150168.1 - 11i/R12: Obtaining Forms Runtime Diagnostics (FRD) In Oracle Applications 11i and Release 12
NOTE:215187.1 - All About the SQLT Diagnostic Tool
NOTE:301137.1 - OSWatcher (Includes: [Video])
NOTE:445166.1 - R12: How To Create An FRD (Forms Runtime Diagnostic) Log Using Forms 10g
NOTE:461053.1 - OSWatcher Analyzer User Guide
NOTE:748642.1 - How to Generate an AWR Report and Create Baselines
NOTE:808005.1 - How to generate pl/sql profiler output for the concurrent program under EBS?
Didn't find what you are looking for?

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=7rljhun1d_34#REF 9/9