MustGather: Performance, hang, or high CPU issues with WebSphere

Application Server on AIX

Troubleshooting

Problem
If you are experiencing performance, hang, or high CPU issues with WebSphere Application
Server on AIX, this MustGather assists you in collecting the data necessary to diagnose and
resolve the issue.

Diagnosing The Problem

Complete the following steps:

(1) Collecting the required data:

a) If verboseGC is not enabled, enable verboseGC

(http://www.ibm.com/support/docview.wss?uid=swg21114927) and restart the server.

b) Download aixperf.sh from this technote: aixperf.sh

(https://www.ibm.com/support/pages/system/files/inline-files/$FILE/aixperf.sh)

The script creates a file named aixperf_RESULTS.tar.gz and ten to twelve javacore text files,
and completes in 4 minutes. As with any script, you might need to add execute permissions
before executing the script (chmod).

c) At the time of the problem, run the script against the problematic process ID (pid)
number(s).  Examples:

If your current process ID (pid) is 23456, then run the command:

./aixperf.sh 23456

If you have multiple pids, that you want to collect data for, then separate them with a space:

./aixperf.sh 23456 9876 4567

** Note: The script does not work using the server name, it must be the current process ID
(pid) number.  

(2) Collecting log files:

Collect the server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log)

from one or more problematic server(s):

a) <profile_root>/logs/server_name/*
b) If you have verbosegc redirected to other files, please send those
as well.

(3) Submitting required data:

Zip the files gathered:

• aixperf_RESULTS.tar.gz created by aixperf.sh

• All javacores (10 to 12) created by aixperf.sh 

• Server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log, verbosegc)

• server.xml for the server(s) that you are providing data for

** Note: The javacores are NOT packed up in the aixperf_RESULTS.tar.gz file so you need to zip those up
and send them.

(4) Send the results to IBM Support: "Exchanging information with IBM Support"
(http://www.ibm.com/support/docview.wss?uid=swg21153852)

Frequently Asked Questions:

• What is the impact of enabling verboseGC?

VerboseGC data is critical to diagnosing these issues. It can be enabled on production systems because it has
a negligible impact on performance (< 1%).

What is the aixperf_RESULTS.tar.gz file and where can I find it?

The aixperf_RESULTS.tar.gz file is created while running the aixperf.sh script and contains output from the
commands called by the script. It is created in the directory from which you execute the script.

What are 'javacores' and where do I find them?

Javacores are snapshots of the JVM activity and are essential to troubleshooting these issues. These files
usually can be found in the <profile_root>. If you do not find the files here, you can search your entire system
for them using the following command:

find / -name "*javacore*"

Only if asked to do so by Support:
The aixmon.sh script collects data every 5 minutes until it is stopped manually. Run the
following command before the issue occurs to start the script:
./aixmon.sh

It creates two files: ps_mon.out and vmstat_mon.out.  The script can be stopped by using
Ctrl+C keys. 

aixmon.sh.zip (https://www.ibm.com/support/pages/system/files/inline-files/$FILE/aixmon.sh.zip)

 
Previous aixperf.sh version:

2019_aixperf.sh (https://www.ibm.com/support/pages/system/files/inline-files/2019_aixperf.sh)

For WebSphere Application Server6.0 and earlier:

./aixperf60.sh 23456 (use the actual pid number for your JVM)

Note: Both aixperf60.sh and dbxtrace_aix.sh are created in the same directory and dbx must
be installed (http://www.ibm.com/support/docview.wss?uid=swg21222456) to diagnose high CPU issues.

aixperf60.sh.zip (https://www.ibm.com/support/pages/system/files/inline-files/$FILE/aixperf60.sh.zip) 
|  dbxtrace_aix.sh.zip (https://www.ibm.com/support/pages/system/files/inline-files/$FILE/dbxtrace_aix.sh.zip)

These scripts will create a file named aixperf_RESULTS.tar.gz and javacore text files. These
scripts need to be executed as the root user. As with any script, you might need to add
execute permissions before executing the script (chmod).

Related Information
Recording your screen to share with IBM Support
(https://www.ibm.com/developerworks/community/blogs/IBMElectronicSupport/entry/recording_your_screen_to_share_with_support?lang=e
n)

MustGather: Read first for WebSphere Application Server (http://www.ibm.com/support/docview.wss?uid=swg21145599)

How to enable verbosegc in WebSphere (http://www.ibm.com/support/docview.wss?uid=swg21114927)

IBM Thread and Monitor Dump Analyzer (http://www.alphaworks.ibm.com/tech/jca)

Submitting information to IBM support (http://www.ibm.com/support/docview.wss?uid=swg21153852)

Steps for getting support for WebSphere Application Server

(http://www.ibm.com/developerworks/websphere/support/appserver_support.html)

Troubleshooting guide for WebSphere Application Server (http://www.ibm.com/support/docview.wss?uid=swg27005324)

Checking if the dbx utility is installed on AIX (http://www.ibm.com/support/docview.wss?uid=swg21222456)

Cross-reference information
Product Component Platform Version

Runtimes for Java Technology AIX All Versions

WebSphere Application Server AIX 9.0, 8.5, 8.0, 7.0, 6.1, 6.0
Product Component Platform Version

WebSphere Application Server - Express AIX 7.0, 6.1, 6.0.2, 6.0.1, 6.0, 5.1

Document Information
More support for:

WebSphere Application Server (https://www.ibm.com/mysupport/s/topic/0TO500000001DQQGA2)

Software version:

9.0, 8.5, 8.0, 7.0, 6.1, 6.0

Operating system(s):

AIX

Document number:

660891

Modified date:

08 February 2022