Copyright (c) 2021, Oracle. All rights reserved. Oracle Confidential.

Oracle E-Business SQL Trace and TKPROF Guide (Doc ID 1674024.1)

In This Document

1 SQL Trace and TKPROF

1.1 Diagnostic Information Available in TKPROF
1.2 Methods for Obtaining SQL Trace in Oracle E-Business Suite
1.3 Common Mistakes
1.4 TKPROF Command
1.5 Parallel Execution Traces
1.6 Parent / Child Traces
1.7 Trace File Location
1.8 Trace File Name
1.9 Raw SQL Trace as well as TKPROF
1.10 Aggregating SQL Traces using trcsess
1.11 TKPROF interpretation
1.12 Runtime Execution Plans / Row Source Operation Counts
1.13 Raw SQL Trace interpretation
1.14 SQLT TRCANLZR Method
1.15 Timed Statistics
1.16 Common issues
1.17 Trace Levels
1.18 Extended SQL Trace
1.19 Incomplete Trace File
1.20 PL/SQL Runtime Dump
1.21 Obtaining Error Message
2 Obtaining Traces (TKPROF) in E-Business Suite
2.1 General Instructions
2.2 Forms
2.3 HTML/Self-Service
2.4 Concurrent Programs - Request Level (for R12 only)
2.5 Concurrent Programs - Program Level
2.6 Profile Option Method
2.7 Enhanced Performance Diagnostic (FND_TRACE)
2.8 Application Module Specific
2.9 Other Forms session (using OAM)
2.10 Concurrent Programs and Forms (using OAM, Debug Workbench)
2.11 Workflow
2.12 Agent Listener
2.13 Mobile Applications (on MWA Server)
2.14 System Level
2.15 From SQL*Plus
2.16 For another database session using DBMS_MONITOR
3 Action Plan Templates
3.1 SQL Trace and TKPROF
3.2 SQLT TRCANLZR Method
Oracle Database Documentation Library Links
Change Log

1 SQL Trace and TKPROF

1.1 Diagnostic Information Available in TKPROF

TKPROF output can be generated from a raw SQL Trace. It formats and summarizes the diagnostic information from the raw SQL
Trace.

This is normally the most important diagnostic resource.

The timed_statistics initialization parameter must be set to TRUE, so that the SQL Trace contains row source statistics (STAT lines) and
CPU times. See Timed Statistics.

The TKPROF is generated from the SQL Trace and reports the contents of the trace. It contains :

The text of the SQLs executed.

 The execution count, elapsed time, CPU time, physical reads (disk), logical reads (query/current) and the number of rows
returned for each SQL. This is further broken down into Parse, Execute and Fetch stages.

The times waited, maximum wait and total waited for each database wait event, both for each SQL and for the whole period of
the trace. It only does this if the level is 8 (with waits) or 12 (with waits and binds). At least level 8 is recommend because it
gives this extra information.

The runtime execution plan for each SQL with the actual row source operation counts for each execution plan line. It only does
this if the row source statistics for the SQL have been written to the trace (See Extended SQL Trace). These statistics will not be
included in any case if timed_statistics is not TRUE.

The execution plan will also contain the actual tables accessed if the SQL uses views or synonyms.

Library cache misses. These are "hard parses"; no execution plan is available in library cache and the optimizer has to determine
an execution plan.

Total SQL and wait statistics for Recursive and Non-recursive SQL Statements.

The database wait event can be used to determine how much time is being spent in

Physical I/O
Logging
Contention
Cluster communication (RAC)
Client-side processing and network communication.

The row source operation counts show the following for each execution plan line:

rows
consistent reads (cr)
physical reads (pr)
physical writes (pw)
time

See Runtime Execution Plans / Row Source Operation Counts.

Level 8 (with waits) is the default choice, because it gives the wait event statistics (Number of Times Waited, Maximum Wait Time and
Total Wait Time) for each SQL.

If the bind values are required then obtain trace at level 12 (with binds and waits). However, this can add significant overhead and
produce a much larger file, particularly if there are a large number of executions (or bulk binding on fewer executions) and/or a high
number of bind variables.

Note that bind values can be viewed in the SQL Trace, but not in the TKPROF.

See the Oracle Database Performance Tuning Guide for more information on SQL Trace and TKPROF.

1.2 Methods for Obtaining SQL Trace in Oracle E-Business Suite

There are a large number of different ways for obtaining SQL Trace and TKPROF in Oracle E-Business Suite.

Forms
HTML/Self-Service
Concurrent Programs - Request Level (for R12 only)
Concurrent Programs - Program Level
Profile Option Method
Enhanced Performance Diagnostics (FND_TRACE) (R12.1.3 onwards)
Application Module Specific
Other Forms Session (using OAM)
Concurrent Programs and Forms (using OAM, Debug Workbench)
Workflow
Agent Listener
Mobile Applications (on MWA Server)
System Level
From SQL*Plus
For another database session using DBMS_MONITOR

1.3 Common Mistakes

To avoid repeating requests for the same information

Specify exactly what is needed in detail and any common mistakes to avoid.
 Review the files as soon as they are received to check that they are correct / complete.

For example :

The TKPROF should be accompanied by the SQL Trace. More

The SQL trace should normally be level 8 (with waits). More

The SQL trace should be complete. More

The SQL trace should be for the correct program and module.

The SQL trace should be from the correct environment.

If the performance issue is for a parallel SQL or DML then SQL Traces for the parent and all of the children will be needed. More

If the performance issue is for a process where the workload is spread across child processes (e.g. Parent and Child Concurrent
Requests) then SQL Traces for the parent and some children (that exhibit the performance issue) will be needed. More

The TKPROF must be generated on the same DB version as the trace (ideally on the same ORACLE_HOME).

1.4 TKPROF Command

The following typical command is recommended for generating TKPROF files:

tkprof <filename1> <filename2> sort= fchela,exeela,prsela

Where <filename1> is the input trace file or a consolidated file of traces produced by the trcsess utility and <filename2> is the file to
which TKPROF writes its formatted output.

In particular:

Sort Options

Use the sort options fchela,exeela,prsela (this is the sum of the fetch elapsed, execute elapsed, and parse elapsed times) to make it
easier to identify which SQL statements are taking most time.

Sometimes it can be useful to have an unsorted tkprof (as well). The SQLs are listed in the order they were parsed. So this can
sometimes help locate the portion of code causing the issue or help identify if trace was disabled (at end of tkprof).

Explain Plan

This should not be necessary. This only generates the explain plan (that would be used) at TKPROF time, it is not the actual execution
plan and can often be different. It also does not have any actual row source statistics (row source operation counts).

It is the actual runtime execution plans (row source operation counts) that are really important; these are stored in the raw trace file
(STAT lines) and automatically reported in the TKPROF file.

Other Arguments

It is not recommended to use any values other then the default for the arguments: waits, print, aggregate and sys. This is because
valuable information could be omitted.

The table, record, insert and width arguments are not usually needed and can be omitted.

See the Oracle Database Performance Tuning Guide for more information on TKPROF.

1.5 Parallel Execution Traces

If a session executes one or more SQL statements in parallel (parallel query, parallel DML or parallel DDL) then it will initiate child
(secondary) processes. If the parent session is being traced then these are traced automatically .

The parent process is called the Query Coordinator (QC) and coordinates all of the child processes and their results.

Note that the not all child processes carry out the same activity and the workload is often not divided out equally. So all of these child
SQL traces will need to be obtained, as well as the SQL trace from the parent session.

The child traces are written to the same location as the parent trace.

See here for trace file location.

See here for parallel child trace file names.

If multiple parallel SQLs are used in a program alongside non-parallel SQL then there could be separate groups of child traces for each
group of consecutive parallel SQLs.

In some instances it can be helpful to aggregate the SQL trace files using trcsess. However, care has to be taken here. Some of the
statistics are sums of all the parallel child processes and can be misleading (appear to be very large).

1.6 Parent / Child Traces

To reduce the elapsed time, some batch processes utilise multiple child processes. For example

A parent concurrent request may spawn multiple child requests.

The autopatch (adpatch) and online patching (adop) tools used on patches and upgrades can use multiple workers (AD
Workers).

Note that all children/workers may not carry out the same activity and that the workload may not be divided out equally.
So, in this case, all child or worker SQL trace files should ideally be obtained along with any parent SQL trace file. As a minimum, SQL
traces for the parent and some children (that exhibit the performance issue) will be needed.

In some instances it can be helpful to aggregate the SQL trace files using trcsess.

1.7 Trace File Location

SQL trace files are written to the directory

<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace

Where <diagnostic_dest> is specified in the initialization parameter diagnostic_dest.

This can be obtained by running the SQL:

SELECT value FROM V$DIAG_INFO WHERE NAME = 'Diag Trace';

See the Oracle Database Performance Tuning Guide and Oracle Database Reference for more information on trace file location and the
initialization parameters above.

Trace File Access Permissions

By default the trace file access permissions are 600 (read and write access for the owner only) for trace files. The owner will be the
database O/S user.

If other O/S users require access to the trace files then set the hidden initialization parameter _trace_files_public=true. This will create
trace files with access permission 644 (read and write access for the owner, read only for the group and others). This is recommended
for E-Business Suite.

The default value for _trace_files_public is false.

If the database server is in a secure environment then this will not weaken external security.

However, consider that there could be secure data contained in bind values within the trace, which will now be available to all O/S
users.

1.8 Trace File Name

The trace file name is normally

<instance>_ora_<ospid>_<identifier>.trc

Where:

<instance> is the database instance name specified by the initialization parameter db_name (v$parameter.db_name).

<ospid> is the OS Process ID identified by the column v$process.spid

<identifier> is the trace file identifier. The identifier will be

The Oracle E-Business Suite user name (e.g. JMACHIN) for SQL traces obtained through Forms or Self Service (OAF).

The user name and request id (e.g. JMACHIN_CR976542) for SQL traces from concurrent programs.

The agent listener component id and timestamp (preceded by the string WFAL) (e.g. WFAL_10002_20130403 for the Workflow
Deferred Agent Listener) for SQL traces from agent listeners.
 The trace file identifier explicitly set (using ALTER SESSION SET TRACEFILE_IDENTIFIER) if SQL Trace is obtained using the
Profile Option method or SQL*Plus.

Alternatively, the OS Process ID (v$process.spid) could be used to identify the trace file or grep can be used on UNIX/LINUX to search
files for key SQL strings or script/module names.

See the Oracle Database Performance Tuning Guide for more information.

Parallel Child Traces

The file name will contain the instance name, parallel child process number and trace file identifier (if it has been set) e.g.

<instance>_p<NNN>_<ospid>_<identifier>.trc

Where <NNN> is the child number, starting from 000. There is a separate process id for each child (and this is different from the
parent process id). The file timestamps can be matched up to locate the correct set of files.

See My Oracle Support document “How to Get 10046 Trace for Parallel Query (Document 1102801.1)”.

If multiple parallel SQLs are used in a program alongside non-parallel SQL then there could be separate groups of child traces for each
group of consecutive parallel SQLs.

Re-used trace file names

If no identifier has been identified (trace is of format sid_ora_pid.trc) then traces from unrelated sessions can sometimes be
concatenated together. This is because the unix process id (pid) is re-used within a short space of time.

This can happen on the middle tier, where there are a few pre-allocated OS processes that get reused among various sessions. Even
though there are different sessions in v$session, several sessions can use one OS process in turn. Since the tracefile_identifier is usually
set, the issue is not normally seen.

The solution is to ask the customer to move (back up) all their existing traces before starting the program(s).

1.9 Raw SQL Trace as well as TKPROF

Always get the raw SQL Trace as well as the TKPROF.

The raw trace itself can be used to get additional information:

Confirmation that the trace is complete or incomplete. More

Errors (err=) and on which SQL (cursor) they occur. More

Specific information on individual execution (instead of aggregates).

Rollbacks or commits and when they occur.

Bind variable values and data types (if level 4 or 12)

RPC procedure (server side) calls from Forms

The time gaps between operations and events. This can help identify where large time gaps (that are unaccounted for on the
TKPROF) occur. Or it could be used to confirm the elapsed time between various points in the SQL Trace. More

Exactly when activities and large individual wait events occur (Wall Clock Time). These can also help identify where client side
processing takes place. More

Database and Host information. More

Session information. More

Module and application context information. More

See Raw SQL Trace interpretation for more information.

The SQLT TRCANLZR Method tool can also be used to analyze traces in more detail.

This gives a detailed breakdown of time spent in different activities. It is also useful for identifying on which main segments (tables)
waits occur and where gaps occur.

1.10 Aggregating SQL Traces using trcsess

Sometimes it can be useful to aggregate several SQL trace files or aggregate content for several trace files. This can be done to obtain:
 Aggregate timings and waits for a whole process

Complete trace information for a single user session on a shared server configuration (connection pooling).

Aggregate timings for a particular module or action

On E-Business Suite this could be :

SQL Traces for multiple child concurrent requests. There are a number of places in E-Business Suite where a parent concurrent
request submits (or spawns) multiple child requests to parallelize a workload.

SQL Traces for multiple AD Parallel workers. The Autopatch (ADPATCH) utility can run some jobs using multiple workers.

SQL Traces for parallel child processes. Care has to be taken here. Some of the statistics (e.g. executions, elapsed time, cpu time
etc) are sums of all the parallel child processes and can be misleading (appear to be very large). The PX waits will also be very
high, as each of the child processes spends a lot of time waiting for other processes (e.g. waiting for the Query Coordinator
(parent) to tell it what to do).

SQL Traces for Self Service (OAF) sessions using connection pooling. There could be several SQL Traces files associated with a
single user session.

Once the output from several SQL Traces has been aggregated, it can be formatted using the TKPROF utility to analyze the information.

The syntax for the trcsess utility is:

trcsess output=<output_file_name> session=<session_id> clientid=<client_id>
service=<service_name> action=<action_name> module=<module_name> <trace_files>

Many of the keywords can be omitted, however one out of session, clientid, service, action, or module must be specified.

For example :

To aggregate all the trace files in the current directory for clientid (user) JMACHIN into one file (summary.trc) :
trcsess output=summary.trc clientid=JMACHIN

To aggregate all the trace files containing the string (tracefile identifier) INVOICE_APPROVAL for service name SYS$USERS
trcess output=invapp.trc service=SYS$USERS *INVOICE_APPROVAL*

To aggregate all the trace files in the current directory for the session with index nn and serial number mmmm
trcsess output=alltraces.trc session=nn.mmmm

The session index and serial number can be obtained from the V$SESSION view.

See the Oracle Database Performance Tuning Guide for more information on trcsess.

1.11 TKPROF interpretation

See My Oracle Support document “TKProf Interpretation (9i and above) (Document 760786.1)”. This gives a very good explanation of
the content of the TKPROF output.

Also see the Oracle Database Performance Tuning Guide.

Note that all times on the Parse/Execute/Fetch and Wait statistics are in seconds. However, the times in the Row Source statistics
(time=) are in micro-seconds (10-6 or 1/1,000,000 of a second).

Note that the EXECUTE statistics for DML statements (e.g. INSERT, UPDATE and DELETE) include the time and statistics for the
modification of Tables and Indexes, but for SELECT they only include the identification of the selected rows. There are no FETCH
statistics for DML statements.

If the initialization parameter timed_statistics is not set to TRUE then the TKPROF will not contain row source statistics (in Row Source
Operation) or CPU timings. These will contain zero values. See Timed Statistics.

Be aware that the physical reads included in the DISK statistics for Parse, Execute and Fetch operations can often be from cache (other
than buffer cache in SGA).

For more information on interpreting the TKPROF see the “What is the challenging SQL? > 1. From Trace” section in My Oracle Support
document "Oracle E-Business Suite SQL Tuning Tips for Customers (Document 2098522.1)".

1.12 Runtime Execution Plans / Row Source Operation Counts

These are very important for determining what part of an execution plan is taking the most time. These can be missing if the trace is
incomplete.
The SQL Trace will only contain row source statistics (STAT lines) if the initialization parameter timed_statistics is set to TRUE. See
Timed Statistics.

For each of the SQL statements in the TKPROF there should be a section with the heading “Row Source Operation” with non-zero row
counts. e.g.

Rows (1st) Rows (avg) Rows (max) Row Source Operation

---------- ---------- ---------- ---------------------------------------------------
2374 2374 2374 SORT ORDER BY (cr=695774 pr=55424 pw=0 time=173895120 us cost=256 size=825 card=25)
2374 2374 2374 FILTER (cr=662233 pr=54443 pw=0 time=391348924 us)
72149 72149 72149 NESTED LOOPS SEMI (cr=117529 pr=35142 pw=0 time=98904258 us cost=179 size=825 card=25)
72149 72149 72149 TABLE ACCESS BY INDEX ROWID PA_EXPENDITURE_ITEMS_ALL (cr=17700 pr=17606 pw=0 time=40884988 us cost=54 size=550 card=25)
72182 72182 72182 INDEX RANGE SCAN PA_EXPENDITURE_ITEMS_N28 (cr=1363 pr=1362 pw=0 time=1095758 us cost=5 size=0 card=115)(object id 1961354)
72149 72149 72149 TABLE ACCESS BY INDEX ROWID PA_COST_DISTRIBUTION_LINES_ALL (cr=99829 pr=17536 pw=0 time=60358713 us cost=5 size=224907672 card=20446152)
72149 72149 72149 INDEX RANGE SCAN PA_COST_DISTRIBUTION_LINES_U1 (cr=87352 pr=5988 pw=0 time=28646560 us cost=3 size=0 card=2)(object id 161390)
69775 69775 69775 FILTER (cr=544704 pr=19300 pw=0 time=71439705 us)
70378 70378 70378 SORT GROUP BY NOSORT (cr=544704 pr=19300 pw=0 time=71360166 us cost=6 size=18 card=1)
337963 337963 337963 TABLE ACCESS BY INDEX ROWID PA_CUST_REV_DIST_LINES_ALL (cr=544704 pr=19300 pw=0 time=66809244 us cost=6 size=18 card=1)
409338 409338 409338 INDEX RANGE SCAN PA_CUST_REV_DIST_LINES_U1 (cr=135464 pr=4969 pw=0 time=23991564 us cost=3 size=0 card=3)(object id 161794)

This is the runtime execution plan; the one that was actually used when the SQL was executed.

Although, this should normally have non-zero row counts, some statements may legitimately have all zero row counts as there are no
rows matching the access predicates.

This section will come after the timing statistics (CPU/Elapsed Time) for the Parse, Execute and Fetch stages in the TKPROF.

This data is obtained from the row source statistics lines that start with the keyword “STAT #n” in the raw trace, where n is the cursor
number they belong to. These can be written to the trace file earlier and more frequently, usually after the results from the first
execution have been fetched, so they are more likely to be in the trace file. See Extended SQL Trace.

If the TKPROF contains no execution plan or only one with the heading “Execution Plan” (as below) then it is likely that the trace file is
incomplete and is missing runtime statistics (STAT lines) for the execution plan lines. See Incomplete Trace Files.

e.g.

Rows Execution Plan

------- ---------------------------------------------------
0 SELECT STATEMENT MODE: ALL_ROWS
0 NESTED LOOPS
0 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF
'FND_PROFILE_OPTIONS' (TABLE)
0 INDEX MODE: ANALYZED (UNIQUE SCAN) OF
'FND_PROFILE_OPTIONS_U2' (INDEX (UNIQUE))
0 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF
'FND_PROFILE_OPTION_VALUES' (TABLE)
0 INDEX MODE: ANALYZED (RANGE SCAN) OF
'FND_PROFILE_OPTION_VALUES_U1' (INDEX (UNIQUE))

This is the explain plan captured when the TKPROF was run and may not be the actual execution plan used at runtime. There is no
guarantee that the two will be the same, especially if the TKPROF is taken on a different environment or the CBO Statistics/Parameters
have changed.

Also see the Oracle Database Performance Tuning Guide for more information.

1.13 Raw SQL Trace interpretation

The lines from the SQL Trace all give information that is summarized and presented in the TKPROF. However, it can be useful to look at
the trace to see more detail or the sequence of the events. The SQL trace also contains extra detail that is not in the TKPROF.

See My Oracle Support document “Interpreting Raw SQL_TRACE output (Document 39817.1)” for more information. This document
explains most of the line types that appear in a SQL Trace and the statistics that are recorded on each line type. It also covers the
syntax of error messages and bind variables.

Also see the Oracle Database Performance Tuning Guide.

Incomplete trace files

See here for information on identifying if a trace file is complete or incomplete.

Time Gaps

If it is not clear from the TKPROF where time has been spent then the timestamp (tim=) can be used to identify time gaps between
lines. The SQLT TRCANLZR Method tool can also be used to identify gaps.

The timestamp can also be used to calculate the elapsed time between various points in the SQL Trace, for example the start and end,
or between the PARSE, EXEC, FETCH, CLOSE operations for a SQL.

The timestamp occurs on all line types except STAT. They are in micro-seconds (10-6 or 1/1,000,000 of a second).
Wall Clock Time

The timestamp can also be used to identify the actual system time (wall clock time) that an event occurred. They refer to a number of
seconds since a reference point. That reference point differs for different platforms and is not particularly relevant. If events during the
trace need to be related to a wall clock time, the first tim= reference in the trace can be equated to a line with the actual system time
(see below) above it in the trace.

Lines that begin with *** contain the system time immediately before the line was output to the trace file in the format YYYY-MM-DD
HH24:MI:SS.nnn. These lines are also output by the Oracle kernel any time that a significant amount of time has elapsed since the last
line output to the trace file.

Large Individual Waits

The elapsed time on waits (ela=) can be used to identify particularly long waits. The wall clock time (above) can also be determined.

This can be particularly useful in identifying where client side processing is taking place ( e.g. "SQL*Net message from client" waits).

Note that this could be waits that are expected (e.g. “SQL*Net message from client” waits at the start and end of the trace, where the
applications tier/client is waiting for user entry).

Database, Host, Operating System, Module and Session Information

At the top of the trace there is a section that contains:

The database version and licensed options

Oracle home path
Operating system (System name)
Host name (Node name)
Operating system version (Release/Version)
Machine type (Machine)
Instance name
Oracle process number
Operating system process id (process pid)
Image name
Session id (SID and Serial)
Service name

It also contains the application context information (set by dbms_application_info). This includes:

The client identifier (CLIENT ID)

Module name
Action name

Note that the following information can be repeated during the trace:

Session Information (e.g. session id).

Application context information (client identifier, module name and action name).

This can be useful if Oracle kernel processes are shared by several Oracle sessions or in connection pooling scenarios. It can also be
useful if a single session progresses through a series of modules/actions.

Errors

The raw trace can be searched for an error message (err=) or a PL/SQL Runtime Dump. The cursor/SQL on which they occur can be
identified.

See here for more information on PL/SQL Runtime Dumps.

See here for information on obtaining error messages.

Be aware of the following

Rollbacks can appear in different formats (e.g. the word “rollback” or the XCTEND line).

Bind positions are offset by 1. BIND#0 refers to the first bind in the SQL (usually :b1)

Timings are recorded in micro-seconds (10-6 or 1/1,000,000 of a second).

Bind and Wait information are only included in the trace for some levels. See Trace Levels.

The cursor number is a unique internal cursor address. It was introduced to save space, so that the SQL text does not need to be
output every time a SQL is executed.
The physical reads statistics (e.g. p= on PARSE, EXEC, FETCH lines and pr on STAT lines) can often be from cache (other than buffer
cache in SGA).

1.14 SQLT TRCANLZR Method

This method analyzes one or more SQL trace files (event 10046) and produces a diagnostic report.

It is similar to TKPROF, but provides extended functionality, such as schema object properties and CBO statistics and other valuable
performance metrics.

SQLT should be installed first on the same database environment where the trace is generated. It can be possible to run on a different
environment (e.g. a test environment), but some of the information (such as CBO statistics) could be misleading.

SQLT is available from My Oracle Support document "All About the SQLT Diagnostic Tool (Document 215187.1)".

The trace files should be in the TRCA$INPUT1 directory (defaults to USER_DUMP_DEST directory during installation).

The ability to analyze multiple SQL trace files is particularly useful if analyzing parallel execution PX traces.

In this case, create a control.txt file with a list of traces (one filename per line and without path specification) and place this control.txt
into the TRCA$INPUT1 or TRCA$INPUT2 directories (default to USER_DUMP_DEST and BACKGROUND_DUMP_DEST respectively during
installation). TRCANLZR will use this control.txt file to find and process the trace files.

To run start SQL*Plus, connecting as the application user that generated the trace, and execute the sqlt/run/sqltrcanlzr.sql script
passing the name of the trace to be analyzed, or the name of the control.txt file populated with filenames. Do not include any path
specification.

See My Oracle Support document "SQLT Usage Instructions (Document 1614107.1)" for more information.

1.15 Timed Statistics

The initialization parameter timed_statistics must be set to TRUE when obtaining a SQL Trace.

If timed_statistics is set to FALSE then the SQL Trace file will not contain the following important information:

Row source statistics (STAT lines)

CPU timings

However, it will still contain waits and binds (if specified in the level or plan_stat argument) and it will still contain count, elapsed time,
disk, query, current and row count statistics for each of the parse, execute and fetch phases.

If the initialization parameter statistics_level is set to ALL or TYPICAL then timed_statistics will be TRUE by default.

Note that timed_statistics cannot be set to FALSE if statistics_level = ALL or TYPICAL.

If statistics_level is set to BASIC then timed_statistics will be FALSE by default, but it can be explicitly be set to TRUE.

See the Oracle Database Performance Tuning Guide and Oracle Database Reference for more information on timed_statistics.

1.16 Common issues

Max Dump File Size

The parameter max_dump_file_size should be large enough to allow the file to be completely written. It is best to set
max_dump_file_size=unlimited.

See Incomplete Trace – Max dump file size has been reached.

File System is full

There should be sufficient disk space available in the file system.

See Incomplete Trace – File system is full.

Online Flows

To ensure complete trace files, sign out of the application after performing the action to be traced.

See Incomplete Trace - SQL Trace disabled before the process has completed.

Pick the trace up and TKPROF after signing out.

See Incomplete Traces - Process is still running when the trace is picked up.

Do not turn trace off before logging out. However, if a self-service flow, using connection pooled sessions, is being traced then trace
can be turned off.

TKPROF Version

The version of TKPROF being used must be the same as the database. If not there can be issues with timing units (which are different
across DB versions).

So, run TKPROF from database node, not the application/concurrent node.

When receiving trace/tkprofs from the customer, always check that the version of the tkprof matches the version of the trace (or the
SQLT / Bug ).

Note that the raw trace can be checked to calculate the time gap between various timestamps and confirm elapsed times.

Trace / TKPROF Environment

When analyzing performance issues, always check that the environment on trace/tkprof matches that on any supporting files (e.g.
SQLT/AWR etc).

Sometimes customers or analysts provide files from an environment that is different from the one where the issue occurred.

TKPROF should be generated on the same environment (as the trace) if using the Explain Plan option; it uses the CBO statistics and
parameters on the environment to generate the execution plan (explain plan).

Connection Pooling / Multiple Traces

Self Service (OAF) uses connection pooling. So there could be several SQL trace files associated with a single user session.

The issue being traced could be in one of the SQL trace files or split across several of them.

See Incomplete Traces – Connection Pooling.

See obtaining SQL Traces in HTML/Self-Service.

Timing Flaws

If there are lots of very short executions then the CPU time can be overstated (accumulation of rounding errors) and can be more than
the elapsed time.

1.17 Trace Levels

By default level 8 (with waits) should be used. The reason for elapsed time can then be analyzed by waits (e.g. Physical I/O,
locks/contention, User action etc.).

Level 12 also gives bind variable information, which can be useful in some circumstances (if the values used in predicates and filters are
needed). However, this can add significant overhead and produce a much larger file, particularly if there are a large number of
executions (or bulk binding on fewer executions) and/or a high number of bind variables. So only use level 12 if binds are needed.
Consider that there could also be secure or sensitive data contained in bind values.

Other levels are

1: Standard SQL Trace

4: Includes Bind Variables

More levels are available in Extended SQL Trace.

1.18 Extended SQL Trace

The row source statistics are written (by default) to the trace file after the first execution of the SQL.

Extended SQL Trace levels control the frequency at which row source statistics are written to the trace file.

They are available as additional levels in the 10046 and sql_trace events. They are also available using the additional plan_stat
argument in DBMS_SESSION and DBMS_MONITOR.

The additional levels are:

Level 16 – STAT lines are written to the trace file for each execution (equivalent to plan_stat=ALL_EXECUTIONS)
 Level 32 - Never write row source statistics (STAT lines) to the trace file (equivalent to plan_stat=NEVER)

Level 64 – (Adaptive) Row source statistics (STAT lines) are written to the trace file for every minute of DB time consumed by
the SQL statement.

Note that there is no level for first_execution, as this is the default (which will be applied at level 8, 12 etc).

The valid values for the plan_stat argument are:

FIRST_EXECUTION – STAT lines are written to the trace file for the first execution only.

ALL_EXECUTIONS - STAT lines are written to the trace file for each execution (equivalent to level 16)

NEVER - Never write STAT lines to the trace file (equivalent to level 32)

Note that lower case values (e.g. all_executions) are also valid.

The adaptive feature (Level 64) is only available on the sql_trace and 10046 events and not on procedures dbms_monitor and
dbms_session.

The advantages of the Extended SQL Trace levels is that the row source statistics are written to the trace much earlier, and not just
after the cursor has closed, so an incomplete trace file is more likely to have row source statistics.

The disadvantage of the default behavior (first_execution) is that the row source statistics are for the first execution only and are
sometimes not representative of all executions. Both all_executions and adaptive write row source statistics on the first execution and
also on all subsequent executions or occasions.

So, in some cases, it may be necessary to set plan_stat to all_executions.

Note that none of these levels (first execution, all_executions, adaptive) will write row source statistics to the trace file before the first
execution and all of its fetches has been completed (this includes bulk collect as well). So incomplete SQL trace files will have row
source statistics (row source operation counts) missing if a long running SQL is still fetching from its first execution.

1.19 Incomplete Trace File

There are various reasons that the trace file is not complete or does not contain the expected information.

Checking the last line (“nnnn elapsed seconds in trace file)” in the TKPROF results is a useful initial check in many of these cases. This
should closely agree with the time observed by users or reported to other logs.

The trace file could be incomplete because:

The process has stopped writing to the trace file because the file system is full.

The process has stopped writing to the trace file because the max dump file size has been reached.

SQL Trace has been disabled before the process has completed.

The process has been killed.

The trace file has been picked up (moved/copied from the dump directory) whilst the process is still running.

The program had an error and quit.

The relevant part of the program ran prior to trace being enabled.

The program uses multiple database sessions (e.g. connection pooling)

The program uses parent / child concurrent requests, multiple workers, Parallel DML or Parallel Query or database links.

File system is full

If the final line of the raw trace file is incomplete then the file system could be full. However, it could be because the trace file has been
picked up (moved/copied from the dump directory) whilst the process is still running.

Max dump file size has been reached

If the raw trace file contains a message “*** DUMP FILE SIZE IS LIMITED TO nnnnnn BYTES ***“ at the end of the file then the max
dump file size has been reached.

The parameter max_dump_file_size should be large enough to allow the file to be completely written. It is best to set
max_dump_file_size=unlimited.

See the Oracle Database Reference for more information on max_dump_file_size.

SQL Trace disabled before the process has completed

If the raw trace file contains a command which disables the trace (e.g. alter session) towards the end and appears to be missing
subsequent SQLs then this could mean that SQL trace has been disabled before the program/session has completed.

Note that if SQL Trace is disabled using other methods such as dbms_session or dbms_monitor procedures then these will not be in the
raw trace file.

So, these situations can be difficult to spot.

Process was killed

This is also difficult to spot, particularly if the process has been killed from the operating system. There could be an error message at
the end of the trace file (e.g. err=1013, which means that the user requested cancel of current operation). There could be waits that
indicate that the client has cancelled the operation (e.g. 'SQL*Net break/reset to client').

Process is still running when the trace is picked up

If the final line of the raw trace file is incomplete then the trace file could have been picked up (moved/copied from the dump directory)
whilst the process is still running. However, it could be because the file system is full.

The program had an error and quit

An error message (err=) close to end of file or a PL/SQL Runtime Dump in the trace indicate that the program quit with an error.

Part of the program ran prior to trace being enabled

This could be the case if the SQL Trace contains some of the later SQL expected, but not the earlier SQL.

Program uses multiple database sessions (Connection Pooling)

In this case there could be several traces associated with a single user session. These can be identified using the tracefile identifier
(often the user id) or using process ids displayed on the screen in Self Service (OAF).

The specific trace file could be identified by searching the traces (using grep in UNIX/LINUX) for key SQL strings or script/module
names. Or the trace files could be aggregated using trcsess.

Alternatively, the particular SQL could be traced manually using SQL*Plus or in-memory runtime statistics could be obtained using the
display cursor report and the ALL+ALLSTATS option. See "Display Cursor Report" and "SQL Monitor Report" sections in the My Oracle
Support document "Oracle E-Business Suite Performance Guide (Document 1672174.1)"

Parallel Execution or Child Requests/Workers

For processes where children or workers are used then all child or worker SQL trace files should ideally be obtained along with any
parent SQL trace file. As a minimum, SQL traces for the parent and some children (that exhibit the performance issue) will be needed.
See Parent / Child Traces.

For Parallel DML and SQL, all of the child SQL traces will be needed as well as the SQL trace from the parent session. See Parallel
Execution Traces.

In both these cases it can be helpful to aggregate the SQL trace files using trcsess.

Row Source Statistics (STAT lines)

Row source statistics are (by default) written to the trace file after the first execution of the SQL. They can be written more frequently
by setting the plan_stat argument to “all_executions” or “adaptive”. See Extended SQL Trace. So incomplete traces are often less of an
issue, although it is still advisable to allow a program to complete and obtain a complete trace.

1.20 PL/SQL Runtime Dump

If a PL/SQL Runtime Dump occurs in a trace then it means an error has occurred. The error will be listed in the dump.

e.g.

*********START PLSQL RUNTIME DUMP************

***Got internal error Exception caught in pfrrun() while running PLSQL***
***Got ORA-28 while running PLSQL
*** PACKAGE BODY APPS.FND_PROFILE: library unit=c0000011fce17ed8 line=264 opcode=0 static
link=800003f84007efa8 scope=1
FP=800003f84007f218 PC=c0000011dfc67a56 Page=0
AP=800003f84007efa8 ST=800003f84007f328
DL0=800003f68014ce20 GF=800003f68014cfc0 DL1=800003f68014cee0
DPF=800003f68014cfa8 DS=c0000011dfc691a8
 DON library unit variable list instantiation
------ ------------ ------------- -------------
0 c0000011fce17ed8 800003f68014cfc0 800003f680135488
1
2
3 c0000011fb899848 800003f680112bf0 800003f680112670
4 c0000011fad1cd08 800003f6801385e0 800003f6801396c0
5 c0000011fce15ab0 800003f680132a08 800003f680132488

1.21 Obtaining Error Message

For runtime dump error messages or err=nnnn error messages in a trace file it is unnecessary to search the Oracle Database Error
Messages manual.

If the DB environment has been set then the following host command can be used.
oerr ora nn

Where nn is the error number.

e.g. for err=28 (which shows that session has been killed).
oerr ora 28

00028, 00000, "your session has been killed"

// *Cause: A privileged user has killed your session and you are no longer
// logged on to the database.
// *Action: Login again if you wish to continue working.

2 Obtaining Traces (TKPROF) in E-Business Suite

This sections contains information for all the tracing methods in Oracle E-Business Suite.

There are some useful screen shots for some of the methods below (Forms, Concurrent Program, Concurrent Request, HTML/Self
Service) in the My Oracle Support document “FAQ: Common Tracing Techniques in Oracle E-Business Applications 11i and R12
(Document 296559.1)”.

General Instructions
Forms
HTML/Self-Service
Concurrent Programs - Request Level (for R12 only)
Concurrent Programs - Program Level
Profile Option Method
Enhanced Performance Diagnostics (FND_TRACE) (R12.1.3 onwards)
Application Module Specific
Other Forms Session (using OAM)
Concurrent Programs and Forms (using OAM, Debug Workbench)
Workflow
Agent Listener
Mobile Applications (on MWA Server)
System Level
From SQL*Plus
For another database session using DBMS_MONITOR

2.1 General Instructions

Here is a list of important information and instructions for all tracing methods :

There should be sufficient disk space available in the file system.

The timed_statistics initialization parameter must be set to TRUE.

The parameter max_dump_file_size should be large enough to allow the file to be completely written. It is best to set
max_dump_file_size=unlimited.

The SQL trace should normally be level 8 (with waits).

If the performance issue is for processes where children or workers are used then all child or worker SQL trace files should
ideally be obtained along with any parent SQL trace file. As a minimum, SQL traces for the parent and some children (that
exhibit the performance issue) will be needed.

If the performance issue is for Parallel DML and SQL then all of the child SQL traces will be needed as well as the SQL trace from
the parent session.
 Allow the actions to complete before disabling trace and picking up the trace files. Ideally log out of the application.

The TKPROF must be generated on the same DB version as the trace (ideally on the same ORACLE_HOME). Do not run from the
application/concurrent node.

2.2 Forms

To enable a SQL Trace for Forms:

1. Log in to the environment that will be traced.

2. Set the profile option “Utilities:Diagnostics“ (internal name DIAGNOSTICS) user-level value to “Yes”.

3. Choose the relevant responsibility for the form and navigate to the form that will be traced.

4. Enable SQL Trace using the menu option : Help > Diagnostics > Trace > Trace with waits. Also select “Unlimited Trace File Size” to
ensure an unlimited size for the trace file.

5. The trace file name and location are displayed in a dialog box. Make a note of the trace file name. This trace will be on the database
tier.

6. Execute the required actions on the Form.

7. When complete, exit the form and log out.

8. Disable SQL Trace using the menu option : Help > Diagnostics > Trace > No Trace

9. Obtain the raw trace file from the trace file location, using the filename that was displayed above when enabling the trace.
Alternatively the trace file identifier (user name), process id or timestamp can be used to identify the trace file.

The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
10. Use the tkprof command to produce a formatted file.

2.3 HTML/Self-Service

This is also sometimes referred to as OAF (Oracle Applications Framework) after the technology framework used on these applications.

To enable a SQL Trace for HTML/Self Service (OAF):

1. Log in to the environment that will be traced.

2. Set the profile option “FND: Diagnostics“ (internal name FND_DIAGNOSTICS) user-level value to “Yes”.

3. Choose the relevant responsibility for the HTML-based application.

4. To enable SQL Trace click the “Diagnostics” icon at the top right of the page > Set Trace Level > Go > Trace with waits > Save.

5. Select “Home” and then execute the required actions.

6. When complete, disable the trace by clicking on the “Diagnostics” icon > Set Trace Level > Go > Disable Trace

7. The “process_ids” will be provided on the left hand side of the screen. Note these down. The tracefile_identifier is the userid.

8. Exit the HTML Application and log out.

9. Obtain the raw trace files from the trace file location, using the process ids that were displayed above, the trace file identifier (user
name) or timestamp.

The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
Self Service (OAF) uses connection pooling. So there could be several SQL trace files associated with a single user session. Either search
the SQL traces to identify the file that the issue is in or aggregate them using trcsess.
10. Use the tkprof command to produce a formatted file.

2.4 Concurrent Programs - Request Level (for R12 only)

To enable a SQL Trace for a Concurrent Program at Request Level:

Note that this method is only available in R12 and 11.5.10 (with latest ATG patch levels).
1. Log in to the environment that will be traced.

2. Set the profile option “Concurrent: Allow Debugging“ (internal name FND_CONC_ALLOW_DEBUG) user-level value to “Yes”.

3. Navigate to the Submit Request window (e.g. Request > Run).

4. Select the Concurrent Request to be run.

5. Click on the “Debug Options” button.

6. Enable SQL Trace by checking the “SQL Trace” Debug Option and then select Debug Option Value "SQL Trace with Waits" (level 8).

Other levels are available (e.g. "SQL Trace with Binds and Waits" (level 12)).

7. Click the “OK” button to confirm the selection. A “Create Debug Rule : Debug Option” page is displayed. Click on the “OK” button
again.

8. Submit the concurrent request.

9. Make a note of the request_id of the concurrent request.

10. Check the progress of the concurrent request in the requests window.

11. When the request has completed, obtain the raw trace files from the trace file location, using the user name and concurrent request
number (which are both contained in the trace file name).

The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
12. Use the tkprof command to produce a formatted file.

Note : This facility is not available for request sets. The “Concurrent Program” or “Profile Option” methods below will need to be used.
If this method does not produce a trace file then check the My Oracle Support document “R12.1.x SQL Trace Does Not Get Created
Using Debug Options from Submit Request Form (Document 1273539.1)”.
2.5 Concurrent Programs - Program Level

To enable a SQL Trace for a Concurrent Program at Program Level:

1. Log in to the environment that will be traced.

2. Choose the relevant responsibility for defining concurrent programs.

3. Navigate to the Concurrent > Program > Define window.

4. Query for the concurrent program to be traced.

5. Turn on SQL Trace for the concurrent program by checking the “Enable Trace” tick box.

Note that this will enable a level 8 trace (with waits) when the program is run.
Timed statistic can be enabled (if timed_statistics != TRUE at system/session level) by checking the “Enable Time Statistics” tick box.
6. Navigate to the Submit Request window (e.g. Request > Run).

7. Submit a concurrent request for the concurrent program.

8. Make a note of the request_id of the concurrent request.

9. Check the progress of the concurrent request in the requests window.

10. When the request has completed navigate to the Concurrent > Program > Define window and disable the trace by unchecking the
“Enable Trace” tick box.

11. Obtain the raw trace files from the trace file location, using the user name and concurrent request number (which are both
contained in the trace file name).

The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
 <diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
12. Use the tkprof command to produce a formatted file.

Note that the profile option “Utilities:SQL Trace” (internal name SQL_TRACE) can be used (by System Administrator) to enable SQL
Trace for all concurrent programs for a user, responsibility, application or site. This is not recommended as it could produce a large
number of trace files, and more specific levels at program and request level are available (above). See the Oracle E-Business Suite
Setup Guide (R12.2 and onwards) or System Administrator's Guide – Maintenance (prior to R12.2) for more information.
2.6 Profile Option Method

If none of the methods above allows the trace to be captured for the required session then all sessions for a particular user can be
traced by setting the profile option “Initialization SQL Statement - Custom” user-level value.

This method also allows the level to be specified. So is often used for concurrent programs where a level 12 trace (binds and waits) is
required.

To enable a SQL Trace using the Profile Option method:

1. Log in to the environment that will be traced.

2. If trace is being enabled for the current (logged in) user account then navigate to the Profile > Personal window, otherwise navigate
to the Profile > System window.

3. Query the profile “Initialization SQL Statement - Custom”(internal name FND_INIT_SQL).

4. Enter the following as the profile value and save.

BEGIN FND_CTL.FND_SESS_CTL('','','TRUE','TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER=''JMACHIN_PERF_TEST''

STATISTICS_LEVEL=ALL MAX_DUMP_FILE_SIZE=unlimited EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 8''');END;

Note: Specify a meaningful identifier. In the example above JMACHIN_PERF_TEST will appear at the end of the trace file name.
Including the user name is recommended.
Note: the quotes in the above statement are all single quotes. Take care that these are entered/pasted correctly into the profile.
Note: If there is existing SQL in the profile then the above should be merged with the existing SQL.
5. Log in (again) to the application using the user account being traced. This is necessary because the Initialization SQL Statement is
only executed at login.

6. Execute the actions to be traced.

7. Exit the application and log out.

8. Navigate to the profile form again, remove the SQL above (ensuring any original SQL remains) from the “Initialization SQL Statement
- Custom” profile and save.

9. Obtain the raw trace files from the trace file location. The tracefile identifier specified above in the Initialization SQL Statement will be
in the trace file name.

The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
10. Use the tkprof command to produce a formatted file.

Note that if SQL trace is enabled for a Concurrent Program (“Enable Trace” tick box filled in the Concurrent Program Define window)
then this will produce a level 8 trace (from 11.5.10 onwards), even if level 12 (binds and waits) has been specified in the profile option
(above). So the “Enable Trace” check box will need to be unchecked for any Concurrent Programs being traced this way.
2.7 Enhanced Performance Diagnostics (FND_TRACE)

From 12.1.3 onwards SQL Trace and other diagnostics (AWR, ADDM, ASH, PL/SQL Profiler, PL/SQL Hierarchical Profiler, Trace Analyzer,
SQLT, Real-Time SQL Monitor Report etc) can be gathered for Oracle E-Business Suite sessions using the Enhanced Performance
Diagnostics (FND_TRACE) feature.

This can be invoked by using the profile option “Initialization SQL Statement - Custom” (FND_INIT_SQL) to call the procedure
FND_TRACE.set_preferences.

To minimize the number and size of traces the profile option should be set at user level and unset immediately after the activity
(causing performance issues) is executed.
For more information and instructions see My Oracle Support document “Enhanced Performance Diagnostics for Oracle E-Business Suite
Release 12.1.3 (Document 1362660.1)”.

2.8 Application Module Specific

Some Oracle E-Business Suite modules have specific methods for tracing some processes. But:

Many of these features are now obsolete.

It is not clear what programs are traced by some others

Some will enable SQL Trace for many different programs and potentially produce a large number of traces.

Some profile options do not provide a SQL Trace but a debug file.

So, it is recommended that other methods above are used instead.

Global Accounting Engine

In Oracle E-Business Suite (Applications) Releases 10.7 and 11 the profile options “AX: Trace Mode” and “AX: Trace Timeout” could be
used to generate SQL Traces. These are obsolete in 11i and R12.

European and Regional Localizations

In Oracle E-Business Suite 11i the profile “Debug: Trace Level for Danish EFT reports” could be used to generate SQL Traces. This is
obsolete in R12.

MRP and Advanced Supply Chain Planning

The profile option “MRP:Trace Mode” may be used to obtain a SQL Trace for some planning processes, such as the Kanban planning
process or Advanced Supply Chain Planning process (in earlier versions).

Order Entry and Shipping

The profile option “OE: Debug Trace” could be used in Oracle E-Business Suite (Applications) Releases 10.7 and 11 to generate a SQL
Trace for a concurrent program. However, in 11i and R12 this feature is obsolete. It is also unnecessary as the concurrent request and
program methods above can be used instead.

Inventory and Warehouse Management

The profile option “INV:Debug Trace” provides a debug facility and does not generate SQL Traces.

Receiving Transaction Processor (Purchasing)

The profile option “PO: Enable SQL Trace for Receiving Processor” will generate a SQL Trace for the Receiving Transaction Processor
(Receiving Open Interface).

It also displays debug information (actions and errors) in the View Log screen.

Note that a SQL Trace can also be enabled for this program using the Concurrent Program, Request or Profile methods above.

Payroll

The TRACE and TRACE level parameters on the table PAY_ACTION_PARAMETERS can be used to generate a SQL Trace file for some
Payroll reports and Pro*C processes. (e.g. Payroll run, PrePayments)

However, SQL Trace can also be enabled for these programs using the Concurrent Program, Request or Profile methods above.

See My Oracle Support documents “Oracle Human Resources (HRMS) Payroll PAY_ACTION_PARAMETERS Comprehensive Overview
(Document 549367.1)” and “How to Produce Trace Files for Oracle Payroll Processes,Reports and Forms (Document 96921.1)” for more
information.

2.9 Other Forms session (using OAM)

From R12 onwards a SQL Trace can be obtained for another Forms Session using the Monitoring feature within Oracle Applications
Manager (Monitoring > Forms Sessions (under Current Activity)).

Click on the Session Details button or the AUDSID to view database information for the selected Forms session. The trace option
(Normal Trace, Trace with Waits, Trace Off, Trace with Binds, Trace with Binds and Waits) can then be specified.
To view the current trace information click View Trace

Once the relevant actions are completed, remember to set the trace option back to “Trace Off” for the selected Forms session.

See the Oracle E-Business Suite Maintenance Guide (R12.2 and onwards) or System Administrator's Guide – Maintenance (prior to
R12.2) for more information.

2.10 Concurrent Programs and Forms (using OAM, Debug Workbench)

In both 11.5.10 (with latest ATG patch levels) and R12 it is possible to SQL trace a concurrent program by creating a Debug Rule from
the Debug Workbench and then choosing Debug Options.

Debug Options include logging, PL/SQL Profiler, SQL Trace and Reports Trace. They also specify the context (either the specific User,
Responsibility or Any (All)) and the number of repetitions or time span for which the debug/trace will be gathered.

The SQL Trace level (regular, with binds, with waits, with binds and waits) can also be specified.

By default the Debug Options are disabled. They can be enabled by setting the profile option “Concurrent: Allow Debugging“ (internal
name FND_CONC_ALLOW_DEBUG) user-level value to “Yes”.

Note that the Debug Options are also available from the Standard Request Submission (SRS) form.

Once the relevant actions are completed, remember to delete or end date the Debug Rule.

See the Oracle E-Business Suite Maintenance Guide (R12.2 and onwards) or System Administrator's Guide – Maintenance (prior to
R12.2) for more information.

2.11 Workflow

Workflow activities are either executed online or in the background. Typically the activities will be executed by one of the following:

Online by the session from where the workflow was started.

Online by the session from where a notification was responded to (not via email).

In the background by the Workflow Background Process

In the background by the Workflow Agent Listener (Business Events)

Workflow activities will be executed online following:

Submission from an online process

Response from a notification (in self service and not email)

Until it reaches one of the following:

An activity that requires input by another user (notification).

An activity that subscribes to a business event that has not been published yet.

An activity is deferred, stuck or timed out. An activity can be deferred if it has a cost greater than the Workflow Engine
threshold.

Note that some workflows have profiles that control whether activities are performed online or in the background (e.g. PO:Workflow
Processing Mode).

Online

Determine how the workflow is submitted. This could be from an Oracle Form, Self Service (OAF) page, concurrent request or another
workflow. A notification will be responded to from a Self Service (OAF) page.

One of the methods above (Forms, HTML/Self Service, Concurrent Program or Concurrent Request) can then be used to trace the
activity.

If the workflow is submitted using a business event then enable SQL Trace for the agent listener.

Background

The workflow background process is actually the concurrent program “Workflow Background Process”. This executes all the background
activities in Workflow.
The Concurrent Programs - Request Level and Concurrent Programs - Program Level methods above can be used to trace this
concurrent program.

Note there could be multiple occurrences of the “Workflow Background Process” concurrent program. These could be for different
Workflow Item Types, cost thresholds and activity statuses (e.g. deferred, timed out and stuck).
See the My Oracle Support document “How To Generate Performance Trace for Workflow Background Process? (Document 734425.1)”,
which also has some useful SQL to identify the trace name.

2.12 Agent Listener

From Oracle E-Business Suite Release 12.1.3 onwards, PL/SQL Agent listeners can be SQL traced at different levels using an internal
Agent Listener parameter SQL_TRACE_LEVEL. Oracle E-Business Suite uses seeded PL/SQL agent listeners for XML Gateway and
Workflow. They are :

10000 ECX Inbound Agent Listener

10001 ECX Transaction Agent Listener
10002 Workflow Deferred Agent Listener
10003 Workflow Deferred Notification
10004 Workflow Error Agent Listener
10005 Workflow Inbound Notifications
10060 Workflow Inbound JMS Agent Listener

The following levels are available :

-1 - SQL tracing is disabled. This is the default.

1 - Normal trace, with no binds or waits
4 - Trace with binds
8 - Trace with waits
12 - Trace with binds and waits
10928 - Trace PL/SQL execution
10941 - Trace name context forever
10053 - CBO Enable optimizer trace

The SQL script afsvcpup.sql is used to change the value of the internal parameter SQL_TRACE_LEVEL for the specified agent listener
service component. However, the agent listener service component (or the container to which it belongs) must be refreshed or
restarted for the change to take effect.

How to obtain SQL Trace for an Agent Listener

To enable SQL Trace for an agent listener, do the following

1. Login to the Application (Middle) Tier as the applications owner.

2. Login to SQL*Plus .

3. From the directory $FND_TOP/sql login to SQL *Plus as user apps.

4. Run the SQL*Plus script afsvcpup.sql, and enter:

The Component ID (listed above) for the agent listener service component that is to be traced.

The Component Parameter ID (that is displayed) for the SQL Trace Level / SQL_TRACE_LEVEL parameter.

The SQL Trace level required (see above). Level 8 (with waits) is recommended.

5. Restart (Stop and Start) the Agent Listener Service Component from the OAM dashboard (Workflow Manager responsibility).

Select “Service Components”, select the relevant Agent listener.

Select 'Stop' from the Actions - wait until the status shows that it has stopped.

Select 'Start' from the Actions List of Value

Note that all PL/SQL Agent Listeners can be re-started by restarting the Workflow Agent Listener Service (OAM -> Site Map > Generic
Services -> View All -> Select -> Generic Service Component Container -> Select “Workflow Agent Listener Service” -> “Restart” Go

6. Once the relevant actions (being traced) have completed, disable the SQL trace using the SQL*Plus script afsvcpup.sql (as above),
specifying level -1 (SQL tracing is disabled) and restart the agent listener as above.

7. Obtain the raw trace files from the trace file location.
The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
The tracefile name will be of the format:

<INSTANCE>_ora_<OS Process ID>_WFAL_<componentId>_<timestamp>.trc

8. Use the tkprof command to produce a formatted file.

For more information see the Oracle Workflow Administrator's Guide and My Oracle Support document “How To Use The New Generic
Service Component Parameter SQL_TRACE_LEVEL to enable SQL Trace for Agent Listener (Document 1448531.1)”.

2.13 Mobile Applications (on MWA Server)

SQL Trace can be enabled on the MWA server for mobile applications (e.g. Warehouse Management and Mobile Supply Chain
Applications).

This can be done by setting the parameter mwa.LogLevel = performance in configuration file (mwa.cfg).

See the My Oracle Support document “SQL*Trace with WMS-MWA: Mobile Applications (Tracing SQL Via Database Trace To Allow For
TKPROF Of Results Of User Actions) (Document 277655.1)” for more information and instructions on how to do this.

Note that one SQL Trace file is written for all mobile user connections.

2.14 System Level

SQL Trace can be enabled/disabled at system level using one of the following :

Set the initialization parameter sql_trace=TRUE/FALSE.

The procedures dbms_monitor.database_trace_enable and dbms_monitor.database_trace_disable

The command “alter system set events ….”

The command “alter system set sql_trace …”

However, this will produce a large number of trace files, and it may be difficult to locate the correct traces.

2.15 From SQL*Plus

Sometimes it may not be possible to generate trace for programs containing problem SQL.

Or a SQL Trace may only be required for individual SQLs, PL/SQL Functions/procedures or just one execution of SQL.

To enable a SQL Trace in SQL*Plus:

This can be done for SQL statements with or without bind variables.
1. Log in to the relevant environment using SQL *Plus as user apps.

2. Run the following SQL to identify the process id for the current session. This may help later in identifying the trace file (although the
trace file identifier should suffice).

SELECT p.spid
FROM v$session s,
v$process p
WHERE s.paddr = p.addr
AND s.audsid = to_number(sys_context('USERENV','SESSIONID'));

3. Check the DB language is correct by running :

SELECT USERENV ('LANG') FROM dual;

4. If the DB language is not correctly set then run:

BEGIN
FND_GLOBAL.set_nls_context (p_nls_language => '<language>',
p_nls_date_format => null,
p_nls_date_language => null,
p_nls_numeric_characters => '.,',
p_nls_sort => null,
p_nls_territory=> null);
END;
/
e.g. For US use <language> AMERICAN.

5. Set the applications context.

The majority of SQLs in E-Business Suite use either:

Multi-org views (11.5.10).

or

Synonyms (R12) that have database policies and contexts applied (virtual private database).

In addition HR uses security groups.

These views and policies rely on the application context (e.g. the user, responsibility, application and security group) to determine the
data returned by views or synonyms. Without the application context being set these views/synonyms will return no rows.

Run the following SQL to identify available responsibilities, applications and security groups for a relevant user.

SELECT u.user_id user_id,

r.responsibility_id resp_id,
r.responsibility_name resp_name,
r.application_id resp_appl_id,
sg.security_group_id security_group_id,
a.application_short_name
FROM fnd_user u,
fnd_user_resp_groups ur,
fnd_responsibility_vl r,
fnd_security_groups sg,
fnd_application a
WHERE u.user_name = '<user_name>'
AND ur.responsibility_id = r.responsibility_id
AND ur.user_id = u.user_id
AND ur.security_group_id = sg.security_group_id
AND a.application_id = r.application_id;

Identify the relevant user_id, responsibility_id, application_id and security group and then run the following to set the applications
context.

BEGIN
fnd_global.apps_initialize(user_id => <user_id>,
resp_id => <resp_id>,
resp_appl_id => <resp_appl_id>,
security_group_id => <security_group_id>);
END;
/

For R12, enable Multi-Org R12 access by running the following :

BEGIN
mo_global.init('&application_short_name',null);
END;
/

Or multi-org access can be simulated by running the following :

BEGIN
mo_global.init('M',null);
END;
/

Or run the following to give access to a single organization (defined in “MO: Operating Unit” profile option) :

BEGIN
mo_global.init('S',null);
END;
/

6. Run the following to set tracefile identifier, timed statistics, max dump file size and trace level:

alter session set tracefile_identifier='<identifier string>';

alter session set statistics_level='ALL';
-- alternatively alter session set timed_statistics=true
alter session set max_dump_file_size='UNLIMITED';
alter session set events '10046 trace name context forever, level 8';

It is important to pick something meaningful and distinctive for the identifier string. This will make it easier to find the relevant trace in
the dump directory.
Timed statistics will ensure that the trace includes row source statistics and CPU times. Statistics_level = ALL (and not just
timed_statistics=true) is recommended because it gives much more information if other diagnostics, such as display_cursor report, are
subsequently used.
Unlimited max dump file size will ensure that the trace file can be completely written.
Trace level 8 (with waits) is normally recommended.
7. Declare any bind variables used by the statement e.g. :

variable B1 number;
variable B2 varchar2(50);

Note: It is important to replicate exactly how the SQL is run in the application. Use the same bind values that would be used by the
application. Do not replace binds with literals.
8. Assign values to binds:

exec :B1 := 123456;

exec :B2 := 'ABC';

Or

BEGIN
:B1 := &B1;
:B2 := '&B2';
END;
/

9. Execute the SQL or PL/SQL actions to be traced.

10. When the SQL, PL/SQL has completed then run the following to disable the trace.

alter session set events '10046 trace name context off';

11. If applicable, de-instantiate all packages in the session and free all memory (e.g. package globals or cached cursors) by running :

exec dbms_session.reset_package;

12. Obtain the raw trace files from the trace file location. The tracefile identifier specified above and the process id identified above will
be in the trace file name.

The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
13. Use the tkprof command to produce a formatted file.

The procedures dbms_session.session_trace_enable and dbms_session.session_trace_disable can be used instead of “alter session set
events …” above.
It is not recommended to use dbms_session.set_sql_trace or alter session set sql_trace to enable/disable SQL trace as no level (e.g.
waits or binds) can be specified.
2.16 For another database session using DBMS_MONITOR

If the trace cannot be obtained by one of the methods above then there are several different ways that the trace can be obtained for
another session using DBMS_MONITOR.

See the following for more information:

The My Oracle Support document “Tracing Enhancements Using DBMS_MONITOR (In 10g, 11g and Above) (Document
293661.1)”

Oracle Database PL/SQL Packages and Types Reference

Oracle Database Performance Tuning Guide

Note that all these procedures enable tracing on the database session only.
Note that SQL Tracing other sessions has limitations.

The APPS user may not have privileges to run DBMS_MONITOR, so it is advisable to run all of the below as SYS.

There are three methods available in DBMS_MONITOR for obtaining a SQL Trace:

Using Client Identifier

Using Service, Module and Action

Using Session ID and Serial Number

Client Identifier: This is normally the user account name (e.g. FND_USER.user_name), however for modules that do not have an E-
Business Suite User (e.g. FNDLOAD or FNDCPASS) it is normally set to 'SYSADMIN'.

Module: This is the name of the currently executing module (application code). This has different formats depending on the type of
program (e.g. concurrent request, Form, Self Service (OAF) page) and on the version of Oracle E-Business Suite. For example:

Concurrent Programs: concurrent program name (internal)

Forms: e:app:frm:<form name>
Self Service (OAF): e.app.fwk:<oaf object>
Concurrent Manager (Standard): e:FND:cp:STANDARD
Concurrent Manager (Inventory): e:INV:cp:INVMGR.

Action: This is the name of the currently executing business action. There are no defined standards for this.

Service: This varies depending on the source of the DB Session and on how the environment is configured.

Database Connection Tagging Feature

The Client Identifier and Service, Module and Action methods rely upon the Database Connection Tagging Feature which records the
client identifier, module and action for a database session. These items are not automatically set for a database session. The
DBMS_APPLICATION_INFO package procedures SET_CLIENT_INFO, SET_ACTION and SET_MODULE are used to explicitly set them.

In Oracle E-Business Suite, the Database Connection Tagging Feature is controlled by the profile option FND_CONNECTION_TAGGING.
To effectively use the DBMS_MONITOR methods client_id_trace_enable or serv_mod_act_trace_enable this should be set to 'Enabled'.

See the Oracle Database PL/SQL Packages and Types Reference for more information on the DBMS_APPLICATION_INFO package.

How to obtain SQL Trace using DBMS_MONITOR

To enable a SQL Trace for another session using DBMS_MONITOR:

1. Confirm or identify the session information (e.g. Client Identifier, Service, Module, Action, Session ID and Serial Number) by querying
the v$session and v$process views.

2. Enable SQL Trace using one of the following methods:

Note that with waits, but not binds is recommended. But other options can be chosen.

Client Identifier
execute dbms_monitor.client_id_trace_enable(client_id =>'<client identifier>',
waits => TRUE,
binds => FALSE);

Service, Module and Action

execute dbms_monitor.serv_mod_act_trace_enable(service_name => '<service_name>',
module_name => '<module_name>',
action_name => '<action_name>',
waits => TRUE,
binds => TRUE);

It is possible to obtain the aggregate trace for all modules or all actions by using the global variables DBMS_MONITOR.ALL_MODULES
and DBMS_MONITOR.ALL_ACTIONS for the module_name and action_name parameters above. Do not use NULL as this will only
aggregate all modules or actions with the value NULL.

Session ID and Serial Number

execute dbms_monitor.session_trace_enable(session_id => <sid>,
serial_num => <serial#>,
waits => TRUE,
binds => FALSE);
If serial_num is NULL but session_id is specified, a session with a given session_id is traced irrespective of its serial number. If both
session_id and serial_num are NULL, the current user session is traced.

3. Allow the required actions to execute on the other session.

4. When they are complete, disable SQL Trace using one of the following methods:

Client Identifier
execute dbms_monitor.client_id_trace_disable (client_id =>'<client identifier>');

Service, Module and Action

execute dbms_monitor.serv_mod_act_trace_disable(service_name => '<service_name>',
module_name => '<module_name>',
action_name => '<action_name>');

Session ID and Serial Number

execute dbms_monitor.session_trace_disable(session_id => <sid>,
serial_num => <serial#>);

5. Obtain the raw trace files from the trace file location.

The trace file name should contain the relevant trace file identifier (user name for a Form/Self Service (OAF) page or user name and
concurrent request id for a concurrent request).

However, it may be necessary to identify the trace file using the OS Process ID (v$process.spid). Alternatively, on UNIX/LINUX, grep
can be used to search files for key SQL strings or script/module names.

The initialization parameter user_dump_dest defines the directory to which the trace file is written. The initialization
parameter<diagnostic_dest> (if set) will define this as follows
<diagnostic_dest>/diag/rdbms/<dbname>/<instname>/trace
6. If tracing a Self Service (OAF) application (using Client Identifier or Service, Module and Action methods) then Self Service (OAF)
uses connection pooling and there could be several SQL trace files associated with a single user session. Either search the SQL traces to
identify the file that the issue is in or aggregate them using trcsess.

7. Use the tkprof command to produce a formatted file.

plan_stat argument

All of the above procedures have an additional plan_stat argument, which specifies the frequency at which row source statistics are
written to the trace. The valid values are 'never', 'first_execution' (default) or 'all_executions'. Uppercase can also be used.

See Extended SQL Trace.

Limitations of SQL Tracing Other Sessions

There are limitations with tracing other sessions if the SQL trace is enabled whilst a SQL is executing, particularly if the SQL is only
executed once.

The trace may not contain information about the SQL (sql_id or sql text) or the Parse, Execute and Fetch statistics. It is likely that it will
not contain Row Source statistics (STAT lines).

If possible, it is better to start the trace before the process or SQL has started.

Alternatively, if you already know the sql_id then use the Display Cursor Report, provided that either:

The session is being SQL traced.

The initialization parameter statistics_level is set to ALL for the session or system.

The initialization parameter "_rowsource_execution_statistics" is set to TRUE for the session or system.

The Display Cursor will not report wait events, but the SQL trace will contain wait events (if waits have been explicitly requested). See
"Display Cursor Report" section in the My Oracle Support document "Oracle E-Business Suite Performance Guide (Document
1672174.1)"
DBA_ENABLED_TRACES view

This view contains information on which client_identifier, service, module and actions SQL Trace is enabled, but only if SQL Trace is
enabled using procedures client_id_trace_enable or serv_mod_act_trace_enable.

Obtaining Session Information

The best way of identifying or confirming the Client Identifier, Service, Module, Action, Session ID and Serial Number is by querying the
v$session and v$process views. e.g.

SELECT s.sid, s.serial#, p.pid, p.spid, p.pname,

s.program, s.client_identifier, s.service_name,
s.module, s.action
FROM v$process p,
v$session s
WHERE s.paddr = p.addr;

Other columns from v$session (e.g. username, sql_id, prev_sql_id) or v$process (tracefile) may be useful too.

Session ID and Serial Number can also be obtained from Help> About Oracle Applications in Oracle Forms.

For a particular concurrent request (request_id) they can be obtained using:

SELECT s.sid, s.serial#, p.pid, p.spid, p.pname,

s.program, s.client_identifier, s.service_name,
s.module, s.action
FROM v$process p,
v$session s
WHERE s.paddr = p.addr
AND p.spid in (SELECT oracle_process_id
FROM fnd_concurrent_requests
WHERE request_id = <request_id>);

For a standard concurrent manager these can be obtained using:

SELECT s.sid, s.serial#, p.pid, p.spid, p.pname,

s.program, s.client_identifier, s.service_name,
s.module, s.action
FROM v$process p,
v$session s
WHERE p.addr = s.paddr
AND s.username like '%APPS%'
AND s.program like '%STANDARD%';

Similarly, the internal concurrent manager, other concurrent managers or transaction managers can be searched for.

e.g. ICM, AMSDMIN, CRM, CRPINQMGR, INVMGR, INVTMRPM, MRPMGR, OAMCOLMGR, PASMGR, PODAMGR, RCVOLTM,
XDP_MANAGER.

Alternatively the v$session.module column can be searched for the same strings (except the internal concurrent manager (ICM)).

DBMS_MONITOR.session_trace_enable in SQL*Plus Current Session or Profile Option Method

It is not recommended to use DBMS_MONITOR.session_trace_enable for tracing the current session in SQL*Plus or in the Profile Option
“Initialization SQL Statement - Custom”.

The APPS user is used in both of these scenarios and it may not have the correct privilege to run DBMS_MONITOR.

3 Action Plan Templates

3.1 SQL Trace and TKPROF

Please provide a SQL Trace (level 8 - with waits, but not binds) and TKPROF.

Please take note of the following:

SQL Trace General Instructions

The SQL Trace can be enabled as follows in:

Forms
HTML/Self-Service
Concurrent Programs - Request Level (for R12 only)
Concurrent Programs - Program Level
Profile Option Method
 Agent Listener
SQL*Plus
For another database session using DBMS_MONITOR

If the customer needs advice on obtaining Session Information for another session then this can be added here.

If the customer needs advice on using trcsess or TKPROF then add here.

3.2 SQLT TRCANLZR Method

Please provide output of SQLT TRCANLZR Method output for the trace files <trace_files>.

The SQLT TRCANLZR output is important because it provides a more detailed analysis of the trace file.

See My Oracle Support document "All About the SQLT Diagnostic Tool (Document 215187.1)" to download the SQLT tool.

SQLT should be installed first on the same database environment where the trace is generated. It can be possible to run on a different
environment (e.g. a test environment), but some of the information (such as CBO statistics) could be misleading.

The trace files should be in the TRCA$INPUT1 directory (defaults to USER_DUMP_DEST directory during installation).

If analyzing multiple trace files (such as parallel execution PX traces) then create a control.txt file with a list of traces (one filename per
line and without path specification) and place this control.txt into the TRCA$INPUT1 or TRCA$INPUT2 directories (default to
USER_DUMP_DEST and BACKGROUND_DUMP_DEST respectively during installation). TRCANLZR will use this control.txt file to find and
process the trace files.

To run start SQL*Plus, connecting as the application user that generated the trace, and execute the sqlt/run/sqltrcanlzr.sql script
passing the name of the trace to be analyzed, or the name of the control.txt file populated with filenames. Do not include any path
specification.

See My Oracle Support document "SQLT Usage Instructions (Document 1614107.1)" for more information.

Oracle Database Documentation Library Links

The documentation libraries are available here.

Reference

19c
12c Release 1
11g Release 2

Performance Tuning Guide

19c
12c Release 1
11g Release 2

PL/SQL Packages and Types Reference

19c
12c Release 1
11g Release 2

Error Messages

19c
12c Release 1
11g Release 2

Change Log

Date Description

1 May 2014 Initial creation.

Added references to My Oracle Support document "Oracle E-Business Suite SQL Tuning Tips for Customers
23 Jun 2016
(Document 2098522.1)" and section numbers.
23 Oct 2019 Updated for 19c. Removed external blog reference. Replaced TRCA with SQLT TRCANLZR.

25 Sep 2020 Updated for 12.2.10. Terminology Updates

Didn't find what you are looking for?