AT Team Hands On Training Package 1

JDK Troubleshooting

Introduction

When a request is made to EBS which needs to process some kind of Java code, it gets routed to
one of the OC4J groups.

There are three different OC4J groups:

"forms" which only processes Form Listener requests
"oafm" which processes any Web Service requests
"oacore" which process everything else, such as OA framework pages

Each OC4J group consists of 1 or more JVM processes, which runs the java code on the Middle
Tier server and returns the results to the client PC

If problems occur when processing these java requests, then we can firstly look at the log files but
for some issues we need to delve deeper into the working of the Java processes.

This activity will focus mainly on this deeper monitoring and analysis

As the majority of the Java processing is done in the "oacore" OC4J group, we will be focusing on
this group.

In Release 12.2, OC4J is replaced by WLS, however the techniques described in task 3 can still be
used

Page 1 v1.0
AT Team Hands On Training Package 1

Objectives
• Be able to identify the current JDK version and understand the steps to upgrade
• Know what diagnostic tools are available and know when and how to use them

Task summary
1. Identify the current JDK version and upgrade to the latest version
2. Modify the runtime parameters for the OACore JDK process
3. Identify and review the OACore log files
4. Extract diagnostic information from the OACore JDK process and review the output

Page 2 v1.0
AT Team Hands On Training Package 1

Task 1 : Identify the current JDK version and upgrade to the latest
version

1. There may multiple Java installations on a server, so is not always clear which version is
being used by EBS. The best way to confirm the java version being used by the Web Node
is to use the script below, which pulls out the java version from the java script that EBS uses
to launch.

Use the following script to confirm the Java version being used by Web Node

#### Start of script

(
sh -c "`awk -F= '$1 ~ /^JSERVJAVA.*$/ {print $2}' $ADMIN_SCRIPTS_HOME/java.sh` -
version;"
) 2>&1 | tee mz_`hostname`_java.txt
#### End of script

HINT : in this case we will see that we are using "java version "1.6.0_07""

2. Review "Overview of Using Java with Oracle E-Business Suite Release 12" (Doc ID
418664.1) and you can see that we can run with JDK 7, which is the latest version.

Review "Using JDK 7.0 Latest Update with Oracle E-Business Suite Release 12.0 and 12.1"
Document 1467892.1 for the steps to upgrade JDK on 10.1.3, 10.1.2 and RDBMS

NOTE : the prereq patches and latest 32bit JDK 7 have been downloaded already for you into
/mnt/stage/EBS and /mnt/stage/java respectively

Copy the patch files from /mnt/stage to the $HOME/patch directory and then unzip and apply as
instructed in the note

3. Go through the steps in Note 1467892.1 to upgrade the JDK in 10.1.3, 10.1.2 and RDBMS
Oracle Home

4. Make sure you test the installation using "Section 5: Verify Application Tier Node
Configurations" and "Section 6: Verify JRE Installation on Database Tier Node" Also
repeat Step 1 (above) to confirm the version now shows as "1.7.0_51"

Page 3 v1.0
AT Team Hands On Training Package 1

Task 2 : Modify the runtime parameters for the OACore JDK process

EBS JDKs are launched with generic tuning parameters. Although these standard parameters are
good for most cases, they can be modified for site specific tuning. These parameters also control
the debug output and other troubleshooting that can be enabled

1. Identify the current settings for your instance.

- First review the $CONTEXT_FILE and look for the variable "s_oacore_jvm_start_options"
which should look something like this:

<oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-server -verbose:gc -Xmx512M

-Xms128M -XX:MaxPermSize=160M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:
+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote
-Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -
Djava.awt.headless=true -Dhttp.webdir.enable=false
-Doracle.security.jazn.config=/oracle/inst/apps/R121PROD_vat01/ora/10.1.3/j2ee/oacore/config/
jazn.xml</oacore_jvm_start_options>

2. Make a change to the parameters

- Edit the CONTEXT_FILE file and change " s_oacore_jvm_start_options"

Change from
value="-server -verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=160M -XX:NewRatio=2 -
XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2

to be
value="-server -verbose:gc -Xmx1024M -Xms128M -XX:MaxPermSize=160M -XX:NewRatio=2 -
XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:
+HeapDumpOnOutOfMemoryError

– Save the changes

– Shutdown EBS Middle Tier server
– Run AutoConfig to propogate the changes to the config files
– Start EBS middle tier services for the changes to take effect
– Run the command " ps -ef | grep oacorestd.out" to check the parameters that are currently in
effect

NOTE: you can also review $INST_TOP/ora/10.1.3/opmn/conf/opmn.conf file to see these runtime
parameters

Page 4 v1.0
AT Team Hands On Training Package 1

3. Review the JDK documentation

Of the parameters explicitly listed, the only ones you would normally ask a customer to consider
modifying are:

-verbose:gc -- Enable verbose messages for Garbage Collection (GC)

-Xmx1024M -- Maximum memory available to the JDK
-Xms128M -- Memory allocated at startup
-XX:MaxPermSize=160M -- Memory allocated to storage of class files
-XX:NewRatio=2 -- subdivision of memory allocated to Java objects
-XX:+PrintGCTimeStamps -- Include timestamps with the GC log information
-XX:+UseTLAB -- Thread local object allocation
-XX:+UseParallelGC -- Enables parallel GC
-XX:ParallelGCThreads=2 -- How many parallel threads to use
-XX:+HeapDumpOnOutOfMemoryError -- as the name implies

These parameters are mostly described in detail in the JDK 7 java documentation
(http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html)

"Non standard" parameters are listed in

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

– Review these documents and find each of the above parameters to confirm their usage

NOTE : as mentioned in the "non-standard" document:

• Options that begin with "-X" are non-standard (not guaranteed to be supported on all VM
implementations), and are subject to change without notice in subsequent releases of the
JDK.
• Options that are specified with "-XX" are not stable and are subject to change without
notice

NOTE : You can also find good references to many of the available parameters in the
"Troubleshooting Guide for HotSpot VM" (http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-
VM/html/index.html) This document is invaluable in researching and troubleshooting Java issues

Page 5 v1.0
AT Team Hands On Training Package 1

Task 3: Identify and review the OACore log files

The log files are in 2 seperate areas :

- OPMN

As OPMN is responsible for starting and stopping JVMs, we see the log files related to the overall
JVM itself in the OPMN area. Specifically:

– $LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
– $LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.out
– $LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err
where X is the JVM number

These log files will show things like garbage collection messages, JVM errors (when crashing),
JVM startup/stopping messages

- J2EE

OC4J runs within the context of a JVM and processes all EBS java requests. Any messages
relating to the OC4J components, or the applications that are running within OC4J, will appear here

– $LOG_HOME//ora/10.1.3/j2ee/oacore/oacore_default_group_X

Although there are a variety of log files, the main ones we are concerned with in EBS are
"application.log" and "log.xml" (if ODL logging has been enabled)

1. Review the above mentioned log files and become familiar with the sorts of messages you
will see

2. Follow the steps in section "Increase OC4J logging for oacore" of "R12, 12.1 - How To
Enable and Collect Debug for HTTP, OC4J and OPMN" (Doc ID 422419.1) which will
increase the OACore logging levels

3. Restart the EBS middle tier OACore processes to enable the changes

4. Login to EBS to generate some activity, then re-review the log files to see the extra detail
that is now being output into these log files

Page 6 v1.0
AT Team Hands On Training Package 1

Task 4 : Extract diagnostic information from the OACore JDK process

and review the output

JDK issues tend to be either performance, JDK crashes or JDK hangs.

For EBS the majority of issues tend to be related to performance. This can be caused by a wide
variety of issues, including things like
• not enough JVMs configured for the number of users
• poor tuning of Garbage Collection (GC) or memory
• JVM thread deadlocks
• poor performance from database queries

There are a wide range of tools available which will allow us to extract and review data that can
help us identify the root cause of any JVM issue. In this task we will take a quick look at some of
these tools

1. Review Garbage Collection (GC) statistics

If a JVM process is taking a lot of CPU and giving poor performance, it is often because the JVM
has run out of memory (controlled by the -xmx setting) We can review the GC data in the log files
to see if there is a lot of "Full GC" which would indicate this could be happening. One possible
way to solve Full GC issues are to increase the memory or to increase the number of JVMs.
However, if the memory is being used due to an application bug (memory leak) then you will soon
get back to the same problem

– Navigate to $LOG_HOME/ora/10.1.3/opmn and review

default_group~oacore~default_group~1.log

--------
14/02/07 14:47:21 Start process
--------
14/02/07 14:47:22 WARNING: ApplicationLogManager is not installed, may result in
loader leaks. Set -
Djava.util.logging.manager=oracle.classloader.util.ApplicationLogManager
1.197: [GC 33024K->10976K(125952K), 0.0476470 secs]
2.050: [GC 44000K->15134K(158976K), 0.0462780 secs]
3.262: [GC 81182K->23799K(158976K), 0.0557210 secs]
4.913: [GC 89847K->32607K(225024K), 0.0612600 secs]
5.924: [GC 164703K->34488K(225024K), 0.0353520 secs]

This output is described very well in http://www.oracle.com/technetwork/java/javase/gc-tuning-6-

140523.html in section titled "Measurement" but really we're looking for lots of "Full GC"
messages to indicate lack of memory

Looking at the GC outputs is a little onerous, so can use third party tools to be able to see the data
more easily

Page 7 v1.0
AT Team Hands On Training Package 1

One useful tool to analyse GC log files is "IBM Pattern Modeling and Analysis Tool for Java
Garbage Collector" which can be downloaded for free from
http://www.alphaworks.ibm.com/tech/pmat

Ordinarily I would download the customers log files onto my local PC and run these tools from my
PC.. however for this task you can run the tool straight from the server GUI screen

– Use your X server session (tightvnc) as it's a GUI tool

– "/mnt/stage/3rdparty/GC/runMe.sh" will launch the tool
– File--> Open Verbose GC file then select
"/oracle/inst/apps/R121PROD_vat01/logs/ora/10.1.3/opmn/default_group~oacore~default_g
roup~1.log"

– you can then use the "Analysis" options to see in table or graph view what the pattern of GC
has been from this file.
– Take some time to review this tool and compare the data shown in the GUI with the raw data
in the log file
– OPTIONAL ACTIVITY : Although written for JDK 5,
http://www.oracle.com/technetwork/java/javase/memorymanagement-whitepaper-
150215.pdf is a detailed look at how GC works. If you get time you can review this
document.

Page 8 v1.0
AT Team Hands On Training Package 1

2. Generate and review Java thread dump

Often it is important to look at what Java is doing within the process. We can ask java to dump out
to a log file it's current state

– Create a Java thread dump

Use the command "kill -3 PID" where PID is your OACore JVM process ID
Wait for about 10 seconds then run the same command again.
Wait another 10 seconds then run the same command, so we have created three thread dumps in
total

The output from the thread dumps are also put into default_group~oacore~default_group~1.log

– Review the file

/oracle/inst/apps/R121PROD_vat01/logs/ora/10.1.3/opmn/default_group~oacore~default_gr
oup~1.log
Search for the string "Full thread dump" and you should find your three thread dumps. The output
is quite small compared to a busy production instance, however is still a long output and is
sometimes difficult to follow

Looking at the Thread Dump outputs is a little onerous, so can use third party tools to be able to see
the data more easily

One useful tool to analyse the Thread Dumps is "IBM Thread and Monitor Dump Analyzer for
Java" which can be downloaded for free from http://www.alphaworks.ibm.com/tech/jca

Ordinarily I would download the customers log files onto my local PC and run these tools from my
PC.. however for this task you can run the tool straight from the server GUI screen

– Use your X server session (tightvnc) as it's a GUI tool

– Copy "/mnt/stage/3rdparty/ThreadDump/*" to your local disk and use " runMe.sh" to launch
the tool
– File--> Open Thread Dumps then select
"/oracle/inst/apps/R121PROD_vat01/logs/ora/10.1.3/opmn/default_group~oacore~default_g
roup~1.log"

Page 9 v1.0
AT Team Hands On Training Package 1

– select one of the three thread dumps and click Analysis--> Thread Detail

– On the right side we see a summary of the threads by status, and can immediately see any
deadlocks. On the left side we see the individual threads, which you can then click on to
see the Java Stack and any threads waiting on it, or it's waiting for
– Take some time to browse the tool and see the information it shows
– Also review the README for this tool, which is located at
http://pic.dhe.ibm.com/infocenter/isa/v4r1m0/index.jsp?topic=
%2Fcom.ibm.esupport.tool.tmda.doc%2Fdocs%2Freadme.htm

Page 10 v1.0
AT Team Hands On Training Package 1

3. JDK command line utilities

There are a variety of command line tools we can use to futher analyse the JVM process. Most are
self explanatory, so will just go through the more useful ones quickly

There is more information available in the JDK 7 documentation at

http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/tools.html and
http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/tooldescr.html#gbdid

– jmap -histo
Run the command "jmap -histo PID | tee mzHisto.txt" where PID is your OACore JVM process ID
We are redirecting the output to a file mzHisto.txt as the output will be quite long. View the
mzHisto.txt file

This output lists the Java objects in memory, listed in order of the amount of memory that is being
taken up. If there is a memory leak, then hopefully this output could give some clues

num #instances #bytes class name

----------------------------------------------
1: 247548 78338376 [B
2: 522785 38198040 [C
3: 43927 31566472 [I
4: 1016559 29127712 [Ljava.lang.Object;
5: 368095 11779040 java.util.TreeMap$Entry
6: 80331 10209728 <constMethodKlass>
7: 317289 7614936 java.lang.String
8: 80331 7074808 <methodKlass>
.............
2500: 1 8 sun.reflect.GeneratedMethodAccessor54
2501: 1 8 sun.reflect.GeneratedSerializationConstructorAccessor8
2502: 1 8 java.rmi.server.RMIClassLoader$2
2503: 1 8 sun.reflect.GeneratedSerializationConstructorAccessor41
2504: 1 8 sun.reflect.GeneratedMethodAccessor4
Total 4247415 269688096

– jmap -heap

Run "jmap -heap PID" where PID is your OACore JVM process ID. This output shows the
memory available in each memory pool within the JDK

– jmap -dump
Run "jmap -dump:format=b,file=mzHeap.out PID"

This will create a (potentially large) file with the state of the JVMs memory at the time of the dump.
This output file can then be analysed off line by several tools. This can be potentially useful to
identify memory leaks

Page 11 v1.0
AT Team Hands On Training Package 1

4. Use Eclipse Memory Analyser to review memory dump file

Using jhat can be a bit tricky, so I prefer using Eclipse Memory Analyser. You can download this
for free from https://www.eclipse.org/mat/

In your X server session (tightvnc). Copy the file

/mnt/stage/3rdParty/MemoryHeapDump/MemoryAnalyzer-1.3.1.20140107-linux.gtk.x86.zip to
your local home directory

Unzip the file, this will install into sub-directory "mat"

cd to "mat" directory and launch the tool with the command "MemoryAnalyzer"

File--> Open Heap Dump then select "mzHeap.out" that you created earlier

When prompted, select Leak Suspects report and click finish

Page 12 v1.0
AT Team Hands On Training Package 1

You can then use the GUI screens to drill down into the heap and quickly find memory leak
suspects. A busier JVM would show more meaningful information, but take some time to
investigate the tool

Page 13 v1.0
AT Team Hands On Training Package 1

5. (OPTIONAL ACTIVITY) Use jhat to review memory dump file

jhat is one tool that allows you to analyse memory dump files, for example a dump file created by
"jmap -dump"

If you get time you can run through this exercise, however I find jhat complex to use compared to
Eclipse. Jhat has the advantage of being readily available, i.e. not needing a 3rd party download.

Run "jhat mzHeap.out" After a short wait, you should see messages as below. Once the "Server is
ready" message shows, you can then review the output

– In your X server session (tightvnc) launch Firefox and launch URL "http://localhost:7000"

Page 14 v1.0
AT Team Hands On Training Package 1

Page 15 v1.0
AT Team Hands On Training Package 1

6. (OPTIONAL ACTIVITY) GUI tools - Jconsole and jvisualvm

Whilst it's nice to see data in a GUI, it's not always practicable to use for EBS Support as we
typically need to gather log files from customers to analyse them. Jconsole and jvisualvm are
however useful tools, so its worth taking a look if you get time.

– Use your X server session (tightvnc) as it's a GUI tool

– In a terminal window, launch jconsole by simply typing in "jconsole"

– By using " $S/adopmnctl.sh status" we can find our oacore JVM process ID, which in my
case is 16245
– Select the local process and click connect
– Take some time to look at the data available in the different tabs
– Login to EBS and navigate through the menus. Check to see if you see any impact to the
Jconsole data.

Page 16 v1.0
AT Team Hands On Training Package 1

– close Jconsole down

– Launch $IAS_ORACLE_HOME/appsutil/jdk/bin/jvisualvm
– Double click your OACore process (16245 in my case)
– Review the overview, then click "Monitor" tab
– Take some time to review the information shown
– In the threads tab, notice you can click the "Thread Dump" button to immediately generate
and review a thread dump

Page 17 v1.0
AT Team Hands On Training Package 1

Page 18 v1.0
AT Team Hands On Training Package 1

Useful References
Guidelines to setup the JVM in Apps Ebusiness Suite 11i and R12 (Doc ID 362851.1)
https://mosemp.us.oracle.com/epmos/faces/DocumentDisplay?id=454811.1

Java SE Technical Documentation

http://docs.oracle.com/javase/

Java 7 documentation
http://docs.oracle.com/javase/7/docs/index.html

JDK Troubleshooting Guide

http://docs.oracle.com/javase/7/docs/webnotes/tsg/index.html

Troubleshooting Guide for HotSpot VM

http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/index.html

Java documentation
http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html

"Non standard" Java HotSpot VM Options

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

Page 19 v1.0