Professional Documents
Culture Documents
IBM JVM 1.4.2 64-Bit in SAP NetWeaver Application Server Java Environments
IBM JVM 1.4.2 64-Bit in SAP NetWeaver Application Server Java Environments
IBM JVM 1.4.2 64-Bit in SAP NetWeaver Application Server Java Environments
IBM Boeblingen Lab, Germany SAP on IBM Systems Development @ SAP Copyright 2010, IBM Corporation
Table of contents
Table of contents .................................................................................................................... 2 Technology Used.................................................................................................................... 5
GC Policy used for SAP NetWeaver Application Server Java................................................................. 5 Important Java VM Parameters and their defaults .................................................................................. 6 Verbose GC output .................................................................................................................................. 8
Tools...................................................................................................................................... 27
Memory Problem Analysis with Eclipse Memory Analyzer .................................................................... 27 Preconditions and Installation ......................................................................................... 27 Configuration ................................................................................................................... 27 Prepare Dump Files for MAT ........................................................................................... 29 What to do with the Dump Files ...................................................................................... 29 Verbose GC Analysis ............................................................................................................................. 30 Thread Dump Analysis........................................................................................................................... 31 IBM Thread and Monitor Dump Analyzer for Java .......................................................... 31 Thread Dump Viewer for SAP NetWeaver Application Server Java 6.40/7.0 ................. 32 SAP Solution Manager - End-to-End Root Cause Analysis .................................................................. 34
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 2
Linux on AMD64/EM64T ........................................................................................................................ 39 Native memory ps......................................................................................................... 39 Linux on Power Systems ....................................................................................................................... 39 Native memory ps......................................................................................................... 39
Resources ............................................................................................................................. 40 About the authors................................................................................................................. 41 Acknowledgements .............................................................................................................. 41 Trademarks and special notices ......................................................................................... 42
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 3
Abstract
Across the IBM platforms J9 (IT4J: IBM Technology for Java) has become the preferred technology to run Java applications in the SAP NetWeaver stack of releases 2004 and 7.0x. On the one hand SAP workload may be specific in some respect. On the other hand a huge quantity of general information is available. Therefore, based on our experience in support and development, this paper aims to provide a compact and relevant selection regarding terminology, description of typical problem situations, tuning options and tool usage. We present selected options and Java parameters and describe some basics of garbage collection. In addition we describe the basic ideas of SAP's default parameterization. Out Of Memory situations and garbage collection issues are the main focus of a section dedicated to problem determination. The documented default parameterization is able to support the whole range of different workloads in the SAP context with few exceptions. If it comes to potential improvements it is key to understand the ideas behind it as well as the characteristic workload of your system. We point to the most important aspects like heap layout and workload profiles to discuss our guidelines. Lastly, we give hands on information regarding tool usage. We describe state of the art tools for analysis of garbage collection and thread dumps. In particular we explain how to prepare your system for use of the Eclipse Memory Analyzer, which stands for a major improvement in memory problem analysis.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 4
Technology Used
SAP NetWeaver 2004 and 7.0x Java stacks run with the IBM Technology for Java VM on IBM platforms. The term Hybrid JVM (J9) is often used, as well as IT4J in the context of IBM i. In the following sections you can find a rather compact overview of technical details of the JVM that are important when it comes to running SAP NetWeaver Java stacks upon IBM Technology for Java. To understand this document a basic understanding of Java technology and garbage collection is required. If you are interested in more technical detail regarding the JVM please consider reading: The IBM Developer Kit and Runtime Environment Diagnostics Guide The IBM Developer Works articles Java technology, IBM style
1
Option
-Xgcpolicy:optthruput (optional) -Xgcpolicy:optavgpause
Description
The default policy. It is typically used for applications where raw throughput is more important than short GC pauses. The application is stopped each time that garbage is collected. Trades high throughput for shorter GC pauses by performing some of the garbage collection concurrently. The application is paused for shorter periods. Handles short-lived objects differently than objects that are long-lived. Applications that have many short-lived objects can see shorter pause times with this policy while still producing good throughput. Uses an algorithm similar to that of the default policy but employs an allocation strategy that is more suitable for multiprocessor machines. We recommend this policy for SMP machines with 16 or more processors. This policy is only available on IBM System p and System z platforms. Applications that need to scale on large machines can benefit from this policy.
-Xgcpolicy:gencon
Subpooling
-Xgcpolicy:subpool
Because of the runtime characteristics of an SAP NetWeaver Application Server Java which tends to create lots of short-lived objects, the generational GC policy should be used. This also offers the best tradeoff between achieving high throughput rates and having short pause times at GC invocations.
Although the diagnostics guide is headlined Java Technology Edition, Version 5.0 it is valid for the IBM (J9, Hybrid) JVM 1.4 as well since this is based on 5.0 technology
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 5
The official SAP parameter recommendations can be found in the following SAP Notes 716927 for AIX 1234382 for IBM i 1319038 for Linux on System z 861215 for Linux on AMD64/EM64T 1406548 for Linux on Power Systems
Here is a list of the most important Java VM parameters -Xgcpolicy:gencon This parameter sets the garbage collection policy. SAP has chosen Generational Concurrent for the reasons given in section Used GC policy for SAP NetWeaver Application Server Java -Xmx This parameter defines the maximum size of the heap storage area which is available for Java objects. 2GB is a good base value with which to start the SAP Java Application Server. However, since application workload can grow, it might become necessary to reserve more memory. Another important point to consider here is to ensure that the complete Java heap is able to reside in main memory. -Xms This parameter defines the initial size of the heap storage area which is available for Java objects. SAP sets this to the same size as the maximum heap size (-Xmx) since this ensures that all memory resources are allocated to the application right at the start and precludes unnecessary operations to re-allocate memory which might be expensive. -Xmn This parameter defines the size of the nursery heap area which is used for allocation of new Java objects. Objects which survive a certain number of garbage collections in this area (tenure age) will be promoted to the tenured area. The tenured area is the remaining Java heap (-Xmx - -Xmn). SAP on AIX and IBM i use a size of 1000 Megabytes, since this has been shown to be a good trade-off in various scenarios e.g. EP or PI .We will refer to this setting in the rest of this paper. For historical reasons other IBM platforms may have different values according to the notes above. -Xverbose:gc This parameter writes the JVM trace information about the garbage collection to the standard output.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 6
-Xsoftrefthreshold This parameter sets the number of GCs after which a soft reference will be cleared if its referent has not been marked. The default is 32, meaning that the soft reference will be cleared on the 33rd consecutive GC in which the referent has not been marked. However, this threshold only applies to the number of global collections which occur rather seldom. Since this has caused problems with some cache implementations in the SAP application, e.g. the Portal Content Directory (PCD) cache, the recommended value for SAP was the aggressive setting of 0 which means soft references are cleared in each global collection but recently changed this to 1 to improve the class/classloader unloading times and to reduce the native memory footprint with SR12. -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+ prepwalk This parameter causes the JVM to write a system dump in case of an OutOfMemory condition. This can be used for the analysis with the Memory Analyzer (MAT). -Xdisableexplicitgc This parameter disables the user/application triggered garbage collections. Invoking such collections could result in unexpected behaviour and degraded performance of the system. -Xj9 (AIX, Linux on System z and Linux on Power Systems) On AIX, Linux on Power Systems and Linux on System z you have to set this option in order to activate the J9 technology. -Xjvm:j9vm23 (Linux on AMD64/EM64T) To activate the 2.3 build of the J9 technology as it is used on AIX and Linux on System z. -Djava.net.preferIPv4Stack=true (AIX only) This parameter helps to avoid startup problems in environments where IPv6 is not configured. -Xthr:minimizeUserCPU (Linux on AMD64/EM64T and Linux on System z) The effect of this setting is minimized user-mode CPU usage in thread synchronization, where possible. The reduction in CPU usage may be a trade-off in exchange for lower performance.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 7
Verbose GC output
If the option -Xverbose:gc is set, the JVM will trace information about the garbage collection to standard output. For SAP systems, this can be found in the files std_<node name>.out or dev_<node name> (e.g. std_dispatcher.out, dev_server0) in the work directory of each instance. The output is XML-formatted. Below you can find sample printouts of the garbage collection traces for the 2 typical types of collections which occur when using generational garbage collection. The collection cycle which occurs most often is the so called scavenger collection, which only collects the nursery area. A typical trace reads:
<af type="nursery" id="412" timestamp="Apr 07 11:46:45 2009" intervalms="3166973.450"> <minimum requested_bytes="240" /> <time exclusiveaccessms="0.234" /> <nursery freebytes="0" totalbytes="362151936" percent="0" /> <tenured freebytes="107362792" totalbytes="402653184" percent="26" > <soa freebytes="107362792" totalbytes="402653184" percent="26" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <gc type="scavenger" id="411" totalid="414" intervalms="3166981.373"> <flipped objectcount="231212" bytes="18608616" /> <tenured objectcount="3871" bytes="343576" /> <refs_cleared soft="266" weak="16498" phantom="0" /> <finalization objectsqueued="2038" /> <scavenger tiltratio="89" /> <nursery freebytes="343468400" totalbytes="362387456" percent="94" tenureage="14" /> <tenured freebytes="106992568" totalbytes="402653184" percent="26" > <soa freebytes="106992568" totalbytes="402653184" percent="26" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="51.178" /> </gc> <nursery freebytes="343466352" totalbytes="362387456" percent="94" /> <tenured freebytes="106992568" totalbytes="402653184" percent="26" > <soa freebytes="106992568" totalbytes="402653184" percent="26" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="59.318" /> </af>
The tag af stands for allocation failure. This means the collection was triggered because a request for a new object allocation could not be fulfilled. One property of af is the timestamp which shows when the allocation failure occurred. The tag minimum requested_bytes contains the size of the allocation which was attempted. This followed by information about the size of and the available space in the nursery and the tenured area before the collection. It is expected that the nursery size will be less than the size you specified with the parameter Xmn. This is because the nursery area is split up into 2 sub-areas: one used for new allocations and the other is used during garbage collections to copy objects which need to be kept but are not old enough to be tenured (the so called flipping). When the JVM starts, the areas are allocated
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 8
half of the nursery size though this ratio might shift. The current ratio is traced in the tag scavenger tiltratio which gives the percentage of the area for object allocations (the part that is accounted for in tag nursery). The tenured area also has 2 sub areas, the small object area (soa) and the large object area (loa) which is usually a lot smaller. Within the tag gc you can find statistics about the collection cycle, including information about the occupancy of the memory areas after collection and about the time it took to perform that collection cycle. After the gc section, again information about the heap occupancy is presented together with the overall time for the gc cycle. For a global collection cycle, the information looks similar:
<con event="collection" id="2" timestamp="Apr 07 11:16:37 2009" intervalms="314604867.541"> <gc type="global" id="3" totalid="413" intervalms="157455948.701"> </gc> </con>
You may find the tag af instead of the tag con and, the gc cycle is not of type scavenger but of type global. You will also find some different statistics information. Since it is difficult to analyze the garbage collection history by reading huge XML files, the tool GCMV is available for parsing and visualizing this data. It is described in section Verbose GC analysis.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 9
GC Problem Determination
Types and root causes of an Out Of Memory (OOM)
Two different types of OOMs can happen. One is the so called real OOM, where a new allocation request can not be satisfied by the JVM. In this case, either the Java heap is completely full or there is not enough continuous memory available to satisfy the allocation request. This is, in most cases, an allocation request for a large object (several MB).
<af type="tenured" id="22" timestamp="Feb 27 06:52:32 2009" intervalms="337.224"> <minimum requested_bytes="588886488" /> <time exclusiveaccessms="0.453" /> <nursery freebytes="209321984" totalbytes="209715200" percent="99" /> <tenured freebytes="431560456" totalbytes="1728053248" percent="24" > <soa freebytes="422116712" totalbytes="1710773248" percent="24" /> <loa freebytes="9443744" totalbytes="17280000" percent="54" /> </tenured> <gc type="global" id="42" totalid="628" intervalms="338.214"> <compaction movecount="8757598" movebytes="770949264" reason="compact to meet allocation" /> <refs_cleared soft="0" threshold="0" weak="0" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="977.783" sweep="11.171" compact="974.564" total="1963.755" /> <nursery freebytes="209660120" totalbytes="209715200" percent="99" /> <tenured freebytes="432168800" totalbytes="1728053248" percent="25" > <soa freebytes="432168800" totalbytes="1728053248" percent="25" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <gc type="global" id="43" totalid="629" intervalms="0.592"> <compaction movecount="11316940" movebytes="1283166184" reason="compact to meet allocation" /> <classloadersunloaded count="0" timetakenms="4.004" /> <refs_cleared soft="12" threshold="0" weak="0" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="1033.621" sweep="6.505" compact="4541.778" total="5586.172" /> <nursery freebytes="209660120" totalbytes="209715200" percent="99" /> <tenured freebytes="432304304" totalbytes="1728053248" percent="25" > <soa freebytes="432304304" totalbytes="1728053248" percent="25" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <nursery freebytes="209660120" totalbytes="209715200" percent="99" /> <tenured freebytes="432304304" totalbytes="1728053248" percent="25" > <soa freebytes="432304304" totalbytes="1728053248" percent="25" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="7551.773" /> </af> JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait. JVMDUMP007I JVM Requesting Snap Dump using '/usr/sap/PRD/D00/j2ee/cluster/server0/Snap.20090227.065239.753836.0010.trc' JVMDUMP012E Error in Snap Dump: {nothing to snap} JVMDUMP007I JVM Requesting Heap Dump using
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 10
'/usr/sap/PRD/D00/j2ee/cluster/server0/heapdump.20090227.065239.753836.0011.phd' JVMDUMP010I Heap Dump written to /usr/sap/PRD/D00/j2ee/cluster/server0/heapdump.20090227.065239.753836.0011.phd JVMDUMP007I JVM Requesting Java Dump using '/usr/sap/PRD/D00/j2ee/cluster/server0/javacore.20090227.065239.753836.0012.txt' JVMDUMP010I Java Dump written to /usr/sap/PRD/D00/j2ee/cluster/server0/javacore.20090227.065239.753836.0012.txt JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".
In such a case the JVM will write a thread dump (javacore), a heapdump and a snap trace. If you are running on SR12 (SR13 for Linux on System z and Linux on Power Systems) or later and the recommended JVM (-Xdump) settings are in place, a system dump will be produced in addition. The failing object cannot be found in the heapdump or system dump, so you have to check within the application, why it is requesting such a large object and if this is the normal application behaviour. To get a first indication you can check the stack trace of the current thread in the javacore. If the application behaviour is normal, then you will have to increase the Java heap, so the JVM can handle these large allocation requests. Some more hints on how to investigate this can be found in the chapter Tuning Recommendations based on Known Application Specific Workload.
The IBM JVM also runs into an OOM, if small allocation requests could be handled but in order to satisfy this request the GC would have to run very frequently. As an indication for this case you will find the following warning ("excessive gc activity detected") in the verbose GC output. It will occur if more than 95% of the time is spent in the GC and the GC is not able to free up more than 3% of the heap.
<af type="nursery" id="322" timestamp="Fri May 23 12:47:49 2008" intervalms="12.459"> <minimum requested_bytes="458768" /> <time exclusiveaccessms="0.479" /> <nursery freebytes="402779848" totalbytes="1179147264" percent="34" /> <tenured freebytes="179642776" totalbytes="2197815296" percent="8" > <soa freebytes="179642776" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <percolating_collect reason="insufficient remaining tenure space" /> <gc type="global" id="133" totalid="375" intervalms="30.736"> <refs_cleared soft="0" threshold="0" weak="0" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="776.538" sweep="42.234" compact="0.000" total="819.292" /> <nursery freebytes="403068104" totalbytes="1179147264" percent="34" /> <tenured freebytes="179642776" totalbytes="2197815296" percent="8" > <soa freebytes="179642776" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <nursery freebytes="403068104" totalbytes="1179147264" percent="34" /> <tenured freebytes="179642776" totalbytes="2197815296" percent="8" > <soa freebytes="179642776" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="837.579" /> </af> <af type="nursery" id="323" timestamp="Fri May 23 12:47:50 2008" intervalms="12.442">
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 11
<minimum requested_bytes="458768" /> <time exclusiveaccessms="0.554" /> <nursery freebytes="403019968" totalbytes="1179147264" percent="34" /> <tenured freebytes="179184008" totalbytes="2197815296" percent="8" > <soa freebytes="179184008" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <percolating_collect reason="insufficient remaining tenure space" /> <gc type="global" id="134" totalid="376" intervalms="30.228"> <refs_cleared soft="1" threshold="0" weak="1" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="552.629" sweep="41.490" compact="0.000" total="594.645" /> <nursery freebytes="403298680" totalbytes="1179147264" percent="34" /> <tenured freebytes="179184008" totalbytes="2197815296" percent="8" > <soa freebytes="179184008" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <warning details="excessive gc activity detected" /> <nursery freebytes="402839912" totalbytes="1179147264" percent="34" /> <tenured freebytes="179184008" totalbytes="2197815296" percent="8" > <soa freebytes="179184008" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="605.693" /> </af> JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait. JVMDUMP007I JVM Requesting Heap Dump using '/usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/heapdump.20080523.124750.2986226.ph d' JVMDUMP010I Heap Dump written to /usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/heapdump.20080523.124750.2986226.phd JVMDUMP007I JVM Requesting Java Dump using '/usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/javacore.20080523.124750.2986226.tx t' JVMDUMP010I Java Dump written to /usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/javacore.20080523.124750.2986226.txt
In this case the JVM will write a javacore, a heapdump and a snap trace. If you are running on SR12 or later and the recommended JVM (-Xdump) settings are in place, a system dump will be produced in addition. The heapdump or system dump can help to analyze why the GC is not able to free up memory to satisfy further allocation requests. In this case the application is holding all the memory, so you have 2 to check if this is the normal application behaviour (i.e. large application footprint ), or if this could be a memory leak in the application. A memory leak will normally show up over a longer period of time, so you will see increasing heap usage over time.
The definition of application footprint in this document is the constantly used memory in the Java heap which survives after a global GC. It is made up of J2EE Framework space and Session space.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 12
This picture shows a reduction of free tenured heap over time even though the GC is running several times the day. The JVM will end up with an OOM due to the memory leak at the end.
Higher application footprints may require to add additional server nodes or to add more Java heap depending on the signature of the workload. Some more hints on how to investigate this can be found in the chapter Tuning Recommendations based on Known Application Workload.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 13
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 14
</gc> <nursery freebytes="838404520" totalbytes="943718400" percent="88" /> <tenured freebytes="552148376" totalbytes="1098907648" percent="50" > <soa freebytes="530170264" totalbytes="1076929536" percent="49" /> <loa freebytes="21978112" totalbytes="21978112" percent="100" /> </tenured> <time totalms="10567.322" /> </con>
It took more than 10 seconds to unload the 37221 class loaders in this example. The largest proportion of the GC time was spent in classloader unloading. This also depends on the system resources (CPU) available for the GC at this time. If the tenure area isnt overused, the interval between the global GCs can be long. The longer the JVM runs without a global GC, the more class loaders will be created. In case of small test/development systems, where the 2GB Java heap is not totally used, you can reduce the Java heap to get a global GC more often. With changing the parameter Xsoftrefthreshold0 to Xsoftrefthreshold1 some caches are working better. As a result less classes and classloaders are reloaded, because they can be reused from the cache. This can reduce the GC pause times too.
<con event="collection" id="18" timestamp="Feb 15 08:52:05 2009" intervalms="220781166.438"> <time exclusiveaccessms="55.381" /> <nursery freebytes="279076808" totalbytes="835537920" percent="33" /> <tenured freebytes="8452320" totalbytes="1098907648" percent="0" > <soa freebytes="8452320" totalbytes="1098907648" percent="0" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <stats tracetarget="605053568"> <traced total="516943032" mutators="505496920" helpers="11446112" percent="85" /> <cards cleaned="625" kickoff="25396506" /> </stats> <con event="remembered set scan"> <stats objectsfound="1658" traced="4699448" timetakenms="17.520" /> </con> <con event="final card cleaning"> <stats cardscleaned="11079" traced="124056016" durationms="2157.260" /> </con> <gc type="global" id="24" totalid="2110" intervalms="220783345.355"> <classloadersunloaded count="22034" timetakenms="417573.054" /> <refs_cleared soft="2784" threshold="0" weak="14622" phantom="0" /> <finalization objectsqueued="4953" /> <timesms mark="82455.598" sweep="5061.014" compact="0.000"
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 15
total="505127.392" /> <nursery freebytes="646404712" totalbytes="835537920" percent="77" /> <tenured freebytes="222339176" totalbytes="1098907648" percent="20" > <soa freebytes="222339176" totalbytes="1098907648" percent="20" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <nursery freebytes="646404712" totalbytes="835537920" percent="77" /> <tenured freebytes="222339176" totalbytes="1098907648" percent="20" > <soa freebytes="222339176" totalbytes="1098907648" percent="20" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="507362.851" /> </con> [Framework -> criticalShutdown] Missed broadcast due to network problems. Restart is required (advanced Reconnect) [Framework -> criticalShutdown] Exiting Listener Loop. This requires a restart of the node. Possible reason is an interrupted reconnect session to the message server. Feb 15, 2009 9:00:39 AM com.sap.engine.core.Framework [SAP J2EE Engine|MS Socket Listener] Fatal: Critical shutdown was invoked. Reason is: Exiting Listener Loop. This requires a restart of the node. Possible reason is an interrupted reconnect session to the message server. Feb 15, 2009 9:00:39 AM com.sap.engine.core.Framework [Thread[Thread-4373,5,SAPEngine_System_Thread[impl:5]_Group]] Fatal: Critical shutdown was invoked. Reason is: Missed broadcast due to network problems. Restart is required (advanced Reconnect) </verbosegc>
The J2EE server node will be restarted automatically after a specific time without a response to the message server (as defined in ms.notification.timeout of the ClusterManager). Make sure that you have sufficient available physical memory in your system. All the memory used by the JVM must be available in the physical memory to guarantee short pause times in the GC.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 16
Frequent GC
Insufficient heap size for the nursery area
If a lot of new objects are allocated in a very short period of time, the objects will be allocated in the nursery area first. If these objects have a short lifetime they will not be moved to the tenure area, so they will die in the nursery area. If this area is too small to handle all these new object allocations the scavenger GC will run with a very short interval.
<af type="nursery" id="177271" timestamp="Feb 15 16:46:31 2009" intervalms="979.280"> <minimum requested_bytes="64" /> <time exclusiveaccessms="0.370" /> <nursery freebytes="0" totalbytes="377487360" percent="0" /> <tenured freebytes="239816480" totalbytes="1728053248" percent="13" > <soa freebytes="239816480" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <gc type="scavenger" id="177262" totalid="177391" intervalms="983.879"> <flipped objectcount="61579" bytes="7634232" /> <tenured objectcount="3086" bytes="225280" /> <refs_cleared soft="5" weak="163" phantom="0" /> <finalization objectsqueued="41" /> <scavenger tiltratio="90" /> <nursery freebytes="369360896" totalbytes="377487360" percent="97" tenureage="14" /> <tenured freebytes="239407144" totalbytes="1728053248" percent="13" > <soa freebytes="239407144" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="84.446" /> </gc> <nursery freebytes="369358848" totalbytes="377487360" percent="97" /> <tenured freebytes="239407144" totalbytes="1728053248" percent="13" > <soa freebytes="239407144" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="89.102" /> </af> <af type="nursery" id="177272" timestamp="Feb 15 16:46:32 2009" intervalms="981.544"> <minimum requested_bytes="64" /> <time exclusiveaccessms="0.477" /> <nursery freebytes="0" totalbytes="377487360" percent="0" /> <tenured freebytes="239407144" totalbytes="1728053248" percent="13" > <soa freebytes="239407144" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <gc type="scavenger" id="177263" totalid="177392" intervalms="986.109"> <flipped objectcount="82457" bytes="11473952" /> <tenured objectcount="2760" bytes="243208" /> <refs_cleared soft="0" weak="305" phantom="0" /> <finalization objectsqueued="12" /> <scavenger tiltratio="90" /> <nursery freebytes="365535200" totalbytes="377487360" percent="96"
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 17
tenureage="14" /> <tenured freebytes="239076584" totalbytes="1728053248" percent="13" > <soa freebytes="239076584" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="105.921" /> </gc> <nursery freebytes="365533152" totalbytes="377487360" percent="96" /> <tenured freebytes="239076584" totalbytes="1728053248" percent="13" > <soa freebytes="239076584" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="111.238" /> </af>
In this case the nursery area (approximately 360MB for new allocations, the rest - up to 400MB - is reserved for the survivor space, depending on the scavenger tiltratio of 90) is filled in less than 1 second (986.109 ms). Only 11473952 bytes are flipped to the survivor space and only 243208 bytes are tenured only, so the most of the new objects are temporary used and die immediately. Another scenario is with large/huge new object allocations as can happen in an SAP PI system. If a large object does not die fast enough, or there are too many new large objects created, the scavenger GC will have to move them to the tenure area. Moving these large objects is an expensive operation. You will also see frequent scavenger GCs but there are more bytes flipped and moved. Verify that this behaviour and workload is expected by the application. If so increase the nursery size to increase the scavenger GC interval correspondingly.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 18
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 19
<traced total="121205584" mutators="121010096" helpers="195488" percent="34" /> <cards cleaned="498" kickoff="14993723" /> </stats> <con event="remembered set scan"> <stats objectsfound="7258" traced="16554728" timetakenms="72.545" /> </con> <con event="final card cleaning"> <stats cardscleaned="57" traced="28147448" durationms="89.847" /> </con> <gc type="global" id="1" totalid="411" intervalms="0.000"> <classloadersunloaded count="7548" timetakenms="1048.516" /> <refs_cleared soft="7291" threshold="0" weak="32395" phantom="0" /> <finalization objectsqueued="9237" /> <timesms mark="101.126" sweep="35.331" compact="0.000" total="1213.780" /> <nursery freebytes="795388248" totalbytes="907292672" percent="87" /> <tenured freebytes="896263904" totalbytes="1098907648" percent="81" > <soa freebytes="841319136" totalbytes="1043962880" percent="80" /> <loa freebytes="54944768" totalbytes="54944768" percent="100" /> </tenured> </gc> <nursery freebytes="795388248" totalbytes="907292672" percent="87" /> <tenured freebytes="896263904" totalbytes="1098907648" percent="81" > <soa freebytes="841319136" totalbytes="1043962880" percent="80" /> <loa freebytes="54944768" totalbytes="54944768" percent="100" /> </tenured> <time totalms="1378.762" /> </con>
<con event="collection" id="9" timestamp="Apr 04 17:18:02 2009" intervalms="25030586.890"> <time exclusiveaccessms="0.288" /> <nursery freebytes="807144" totalbytes="905299968" percent="0" /> <tenured freebytes="13254424" totalbytes="1098907648" percent="1" > <soa freebytes="658104" totalbytes="1079128064" percent="0" /> <loa freebytes="12596320" totalbytes="19779584" percent="63" /> </tenured> <stats tracetarget="190257376"> <traced total="123853808" mutators="86309416" helpers="37544392" percent="65" /> <cards cleaned="556" kickoff="8113332" /> </stats> <con event="remembered set scan"> <stats objectsfound="7229" traced="15863280" timetakenms="51.182" /> </con> <con event="final card cleaning"> <stats cardscleaned="152" traced="21589528" durationms="57.660" /> </con> <gc type="global" id="14" totalid="6567" intervalms="8594942.868"> <classloadersunloaded count="8860" timetakenms="1122.505" /> <refs_cleared soft="9138" threshold="0" weak="35746" phantom="0" /> <finalization objectsqueued="11663" /> <timesms mark="144.068" sweep="66.509" compact="0.000" total="1336.675" /> <nursery freebytes="802748896" totalbytes="905299968" percent="88" /> <tenured freebytes="881640008" totalbytes="1098907648" percent="80" > <soa freebytes="864702200" totalbytes="1079128064" percent="80" /> <loa freebytes="16937808" totalbytes="19779584" percent="85" /> </tenured> </gc> <nursery freebytes="802748896" totalbytes="905299968" percent="88" />
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 20
<tenured freebytes="881640008" totalbytes="1098907648" percent="80" > <soa freebytes="864702200" totalbytes="1079128064" percent="80" /> <loa freebytes="16937808" totalbytes="19779584" percent="85" /> </tenured> <time totalms="1447.785" /> </con>
In this example the application footprint is nearly constant. The tenure area is filled several times and after a global GC there is around 80% free (about 840MB of 1048MB tenure area). The next graph shows this from another perspective, where you can easily see the used tenure memory after a global GC over time:
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 21
To let the GC work optimally some extra memory in addition to the application footprint is needed; otherwise a global GC will happen too often. If the majority of short-lived objects can be handled in the nursery, a small extra amount of memory is necessary for the tenure area, in addition to the application footprint for the tenure area. Experience has shown that the tenure area should be at least 20-30% larger than the application footprint. With the standard SAP parameter recommendation the system can accommodate an application footprint of around 700-800MB without any problems. In this example the footprint is much smaller, around 340MB, so there is a big reserve for some peak workloads. There should be enough space to handle occasional large object allocations, otherwise the JVM will run into an OOM and the application will be restarted.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 22
With very small application footprints, e.g. in test or development systems you can also think about reducing the tenure area and so the total heap. This will fill up the tenure area faster, so a global GC will happen more often. As the heap is smaller the GC pause time will be shorter. In addition the number of class loaders and classes are lower. As less of them are created in a shorter period of time, the class loader unloading will be done faster. User interactive environments will benefit from these shorter GC pause times with a more constant response time and the overall memory usage in the system will be reduced. Be careful about reducing the heap to less than 2GB, because some reserve for peak workload is needed, otherwise you risk an OOM and JVM restart.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 23
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 24
The nursery size was set to 7GB and the tiltratio is at 50% (3.5GB allocation and 3.5GB survivor), so these huge allocation requests (objects of 500MB size) can be handled in the nursery area. There is no need to move them to the tenure area, as they are all dying in the nursery area. As a result there was no global GC for 48 hours, even though the tenure area was only 1GB in size. This is a very specific case, however, in with such a large nursery the tenure area should be increased as well. Experience has shown that an increase of the nursery should be normally be accompanied by increase of the tenure area. To handle large objects it would make sense to increase the nursery area to 2GB for example. For peak workloads which can not be handled in the nursery area, the tenure area should be increased to 2GB too, so the total heap size will be 4GB. The larger nursery leads to longer scavenger GC pause times. This is therefore only an option for non user-interactive applications, like SAP PI, where we are looking for throughput and not for short response times. To get shorter scavenger GCs (nursery GC), the nursery size may be reduced if the number of shortlived objects isnt too high. Check the tilt ratio, to get a feeling for how many objects survive and are copied from the allocation to the survivor area in the nursery. The GC adapts the tilt ratio from 50% up to 90%, depending on the amount of new allocation and surviving objects. User interactive applications like SAP Enterprise Portal will benefit from short GC pause times; provide short and constant user response times. The freed up memory can be used for the tenure area if the application footprint is larger then 1GB without using more memory in total.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 25
With the standard SAP parameter recommendations you should see up to 1GB of native heap under real heavy workload on the system. The total (Java heap and native heap) must, therefore, be considered for the system sizing; otherwise the system might end up paging with resulting bad overall performance. You can find very detailed articles at developerWorks about native memory on AIX: Thanks for the Memory- Understanding how the JVM uses native memory on AIX and for Windows and Linux: Thanks for the Memory- Understanding how the JVM uses native memory on Windows and Linux For monitoring of the native memory usage; have a look to the Appendix.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 26
Tools
Memory Problem Analysis with Eclipse Memory Analyzer
Memory analysis of a SAP NetWeaver Java stack has in general to deal with very large heaps and a huge amount of objects. Many of the existing tools were unable to handle dumps created in such environments efficiently. With the Eclipse Memory Analyzer, aka Memory Analyzer Tool (MAT), there is a tool at hand that can address this issue. Making use of the so-called dominator approach [cf. SDN Blog Dominators in the SAP Memory Analyzer] to parse and sort objects it can help to investigate productive heaps in a reasonable amount of time and on a relatively small PC. There is no point in reproducing the excellent MAT documentation here, as it is available in several places. Entry points for generic information, necessary downloads and detailed installation instructions, are available on to the SDN Wiki page Java Memory Analysis and the MAT Eclipse page. This paper points out the specifics regarding J9. It provided information about installation packages, configuration of your system and gives a description of how to provide dumps.
At the server side your JVM will need to be at minimum service release level SR12 (SR13 for Linux on System z) in order to trigger dumps with J9 which can be parsed by MAT.
Configuration
Note that information regarding configuration and preparation of dump files for MAT might depend on the platform. If you are not sure, you should always be able to find the required information in the relevant SAP Notes 1259465 for AIX 1267126 for IBM i 1263258 for Linux on AMD64/EM64T 1336952 for Linux on System z 1265455 for Linux on Power Systems
With Java parameter Xdump you can register agents in the VM which will generate dumps conditionally. Those agents are cumulative and do not mutually exclude each other. From our experience, they do not impact system performance significantly.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 27
You should certainly add an agent for OutOfMemoryError conditions in the server processes, because this is the most critical case for which you would want be able to analyse the memory usage of your applications: -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+prepwalk In some cases you may want to trigger dumps on request. This has to be handled with care, though: Generation of every single dump will require significant resources in terms of performance and disk space: -Xdump:system:events=user,request=exclusive+prepwalk+compact User event in this context means a signal SIGQUIT, which is sent to the respective process. This can be done in several ways, e.g.: kill -3 <pid> call jcmon and choose options 20 (Local Administration) 10 (Trigger dump) <node> Dump Stack option in the process menu of sapmmc (http://<host>:5<inst>13)
Naming conventions and location of the dumps may differ from platform to platform. Look up the active defaults by calling java Xdump:system:defaults Defaults can be influenced in various ways. If you want to choose your own format or need to route the dumps into a specific file system you can for example specify the new name pattern as an additional parameter e.g. -Xdump:system:defaults,file=/my_file_system/ core.%Y%m%d.%H%M%S.%pid.%seq.dmp Or as extension to the options above e.g. -Xdump:system:events=user,request=exclusive+prepwalk+compact,file=/my_file_system/ core.%Y%m%d.%H%M%S.%pid.%seq.dmp If the given path is not an absolute path the working directory of the process will be the dump location. This can be overruled by setting environment variable IBM_COREDIR accordingly. The same methods hold for thread dumps and heap dumps, i.g.: -Xdump:java:defaults,file=javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:defaults,file=heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd The relevant environment variables would then be with IBM_JAVACOREDIR and IBM_HEAPDUMPDIR, respectively.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 28
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 29
Verbose GC Analysis
Analysis of the heap with MAT tends to be relevant for exceptional and problematic cases or for application development. Complementary, though of a more administrative nature, is the task of monitoring how heap usage develops over time and checking for indications of any pathological behaviour in the VM. This can be addressed by observing garbage collection, and therefore, Java parameter -verbose:gc is set by default. As garbage collection occurs very often during lifetime of a server process, it is exhausting, if not impossible, to get a clear picture by looking at the pure text. Also the amount of information available for a single collection can also be quite complex and, therefore, a reasonable analysis will require some tool for assistance. The current tool of choice is the IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer (GCMV). The snapshots in the discussions above were made with GCMV. GCMV is a plugin to the IBM Support Assistant (ISA). Note that you will have to install at least one product like IBM Developer Kit for Java 1.4 within ISA, to enable GCMV within the tool. Entry point for downloads is http://www.ibm.com/software/support/isa/. Tool handling is menu-based and quite intuitive. Besides the line plot, a summary report with observations and recommendations is given, reflecting the chosen observables. Be aware, though, that the recommendations are based on very generic assumptions and are, in many cases, irrelevant for the SAP Java environment with its very specific work load. The IBM Pattern Modelling and Analysis Tool for Java Garbage Collector (PMAT) is easier to install but has less sophisticated functionality. It is available with invocation instructions at http://alphaworks.ibm.com/tech/pmat/download.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 30
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 31
Thread Dump Viewer for SAP NetWeaver Application Server Java 6.40/7.0
The Thread Dump Viewer for SAP J2EE Engine 6.40/7.0 which is attached to SAP note 1020246 as jar file tdv640.JAR, may be used as an alternative to the tool described above. It can be invoked the same way: java -jar tdv640.JAR
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 32
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 33
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 34
Summary
The documented default parameterization is able to support the majority of different workloads in the SAP context. Some exceptions may require specific changes. This document will assist in detection of the root cause of problems and in finding the right tuning for the system workload. The following points must always be considered: 1. Do not tune the settings only for the current specific workload: there should be room for some workload peaks. 2. System stability is more important than system performance. For user interactive applications, like the SAP Portal, it is better to increase the number of server nodes instead of increasing the heap size per server node for load balancing and availability reasons. In cases of very large message transfer (huge object allocations), e.g. in SAP PI, this wont help to solve an OOM situation. The message has to fit in the Java heap of one J2EE server node. In this case, the Java heap size must be increased as described in the tuning recommendations. For the overall system sizing, the native heap must be added to the Java heap: otherwise the system can run into paging and as a result the Java performance will drop dramatically.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 35
Appendix
Platform specifics
AIX
Native memory svmon
To monitor the native memory of the JVM (jlaunch) you can use the AIX command svmon. For a first overview use the script which is provided with the IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer (GCMV). The script and a description of use can be found via Help -> Help Contents -> IBM Monitoring and Diagnostics Tools for Java Garbage Collection and Memory Visualizer -> Supported data types -> Native memory -> AIX native memory. Run the script against the PID of the SAP J2EE node (search for the jlaunch named processes with the ps ef command) over a longer period of time (several days). The collected data can be loaded into the GCMV and will show you the overall memory usage (Java heap and native heap) of the specific process graphically.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 36
IBM i
Memory pools
As mentioned earlier it is key to avoid paging in Java processes. On IBM i it is possible to route certain processes into a dedicated memory pool. This way sufficient memory may be assigned to Java processes and memory usage of different workload as e.g. ABAP and Java may be separated. Utilizing memory pools has provenvery valuable in this regard. SAP note 1023092 describes in detail how this can be setup.
WRKJVMJOB
Whenever a quick snapshot of a Java process is required the OS command WRKJVMJOB can help. It yields quick access to all processes that loaded an IT4J JVM. For every process GC, thread, environment and other job information can be accessed. Within GC information the native memory consumption can be observed as Other memory.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 37
Linux on System z
Native memory ps
To monitor the native memory of the JVM (jlaunch) you can use the Linux command ps. The procedure for Linux on System z is very similar to AIX. For a first overview use the script which is provided with the IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer (GCMV). The script and a description of use can be found via Help -> Help Contents -> IBM Monitoring and Diagnostics Tools for Java Garbage Collection and Memory Visualizer -> Supported data types -> Native memory -> Linux native memory. Run the script against the PID of the SAP J2EE node (search for the jlaunch named processes with the ps ef command) over a longer period of time (several days). In some circumstances this script doesnt work correctly. In this case please use the following script:
#!/bin/sh # The process id to monitor is the first and only argument. PID=$1 # The interval between ps invocations, in seconds. INTERVAL=3 # Echo the date line so we know when monitoring began. echo timestamp = `date +%s` # Echo the interval frequency. echo "ps interval = $INTERVAL" while ([ -d /proc/$PID ]) do ps -p $PID -o pid,vsz,rss sleep $INTERVAL done;
The collected data can be loaded in to the GCMV and will show you the overall memory usage (Java heap and native heap) of the specific process graphically.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 38
Linux on AMD64/EM64T
Native memory ps
To monitor the native memory you can use the same approach as for Linux on System z.
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 39
Resources
These Web sites provide useful references to supplement the information contained in this document: Java technology, IBM style: Garbage collection policies, Part 1 http://www.ibm.com/developerworks/java/library/j-ibmjava2/index.html Java technology, IBM style: Garbage collection policies, Part 2 http://www.ibm.com/developerworks/java/library/j-ibmjava3/index.html Understanding how the JVM uses native memory on AIX http://www.ibm.com/developerworks/java/library/j-nativememory-aix Understanding how the JVM uses native memory on Windows and Linux http://www.ibm.com/developerworks/java/library/j-nativememory-linux DTFJ adaptor available at developerWorks http://www.ibm.com/developerworks/java/jdk/tools/mat.html ISA download site http://www.ibm.com/software/support/isa/ Java diagnostics, IBM style, Part 2: Garbage collection with the IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer http://www.ibm.com/developerworks/java/library/j-ibmtools2/index.html PMAT download site http://alphaworks.ibm.com/tech/pmat/download IBM Thread and Monitor Dump Analyzer for Java http://www.alphaworks.ibm.com/tech/jca SDN Blog Dominators in the SAP Memory Analyzer https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/6993 SDN Wiki page Java Memory Analysis https://www.sdn.sap.com/irj/scn/wiki?path=/display/Java/Java+Memory+Analysis MAT Eclipse page http://www.eclipse.org/mat MAT download site http://www.eclipse.org/mat/downloads.php: SAP NetWeaver Extensions for MAT https://wiki.sdn.sap.com/wiki/display/Java/Download+the+SAP+Netweaver+enhancements+for+ the+Eclipse+Memory+Analyzer SAP Service Marketplace (SAP Notes) https://service.sap.com/notes SAP End-to-End Root Cause Analysis http://service.sap.com/diagnostics
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 40
Acknowledgements
Reviewer:
Enrico Mayer Harald Duvenbeck Bernd Franz Sally Power Kolby Hoelzle Oliver Stabel
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 41
Version 3/25/2010
http://www.ibm.com/support/techdocs
IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments
Page 42