BasicsA Java dump, also known as a Java core, Java thread dump, or a thread dump is a file that contains the following sections:
- All of the threads that run on a Java Virtual Machine (JVM).
- All of the monitors on a JVM.
- Some useful information about the system that the JVM runs under.
Format – Dump Routines
The following picture is a representation of a Java dump broken into subcomponents:
The following list describes the subcomponents found in Figure 1:- Tags – The tags are located on the left side of the Java dump. They help you visualize the subcomponent represented by the data on the right side of the file. For example, 2LKINFLATEDMON is a tag for the LK subcomponent.
- TITLE – Describes what causes the JVM to produce the Java dump file. The TITLE subcomponent contains the date and time of occurrence and the location in your file system. You can use this information to determine if the Java dump was created by a user or was system-generated.
- XHPI – When available, this dump routine describes the operating environment, memory information, user limits, and other operating system related details.
- CI – Use this section to identify the JVM build, the class path that the JVM uses, the JVM system property variables, and other system information related to the JVM.
- LK – Provides information for all of the monitors in the JVM. You can use this section to analyze how resources are used by various threads.
- XM – This section provides a detailed list of all the threads and their states. It also provides the current stack trace for each thread listed. You can use the stack trace to relate the current activity of thread with the source code.
- CL – Lists all of the classes and class loaders that load at run time. If you know the classes or libraries required by your application, you can cross-reference the information against the list of classes provided in this section.
- JVMMI DC DG ST XE – See the Appendix for more details on these sections.
Operating System Differences
Java dumps vary from one operating system to another. Although the variation doesn’t differ greatly, most of the differences are found in the operating-specific information that the Java dump routines collect. One of these differences is how the environment variables that are set on a specific operating system in a Unix system, for example, are handled differently than Windows environment variables. These variables will display differently when you collect a Java dump on a Unix system and compare it to one collected on a Windows system.
Java Service Release Differences
Between each Java service release, the Java dump routine is enhanced to provide the most information possible when diagnosing problems.
For this reason, you’ll notice that a Java dump from one service release of the JDK to another is formatted differently in terms of tabs or spaces or additional information. For example, if you compare JDK 1.2.x Java dumps to JDK 1.3.x Java dumps, you’ll notice that the structure of the two documents is very different. Some data that’s found on the JDK 1.2.x Java dump is rearranged in the JDK 1.3.x Java dumps, such as the XM dump and LK dump routine locations.
Java Dump – Thread & Monitor Analysis
The following is an example of a thread stack trace in the XM dump routine:
3XMTHREADINFO “Thread-6” (TID:0x181D6A0, sys_thread_
t:0x133AA648, state:R, native ID:0x948) prio=5
4XESTACKTRACE at java.net.PlainSocketImpl.
socketAccept(Native Method)
4XESTACKTRACE at java.net.PlainSocketImpl.
accept(PlainSocketImpl.java:446)
4XESTACKTRACE at java.net.ServerSocket.
implAccept(ServerSocket.java:264)
4XESTACKTRACE at java.net.ServerSocket.
accept(ServerSocket.java:243)
4XESTACKTRACE at com.ibm.disthub.impl.multi.server.
SSConnMgr.threadMain(SSConnMgr.java:251)
4XESTACKTRACE at com.ibm.disthub.impl.client.
DebugThread.run(DebugThread.java:66)Example 1 – This is an example of a Java thread’s stack trace in the XM Dump Routine The first line of the previous example indicates that there’s a thread named Thread-6 that’s in a runnable state as indicated by the “state:R” parameter. This means that Thread-6 is either running or ready to run. Like all threads, Thread-6 has two primary identifiers: TID and sys_thread_t. For Thread-6, the values include 0x181D6A0 and 0x133AA648 respectively. You can cross-reference these values with the monitor data in the LK dump routine to detect deadlocks and identify problematic threads.
The remaining lines of text, identified by the “4XESTACKTRACE” tag, indicate the sequence of methods performed by the thread when the Java dump occurred. It is like a standard Java stack trace where the beginning of the code execution is read from the bottom to the top.
Just In Time (JIT) compiling is enabled when line numbers are replaced by the keywords “Compiled Code” in a stack trace as shown in Example 2. Disable JIT compiling to display the line numbers. Refer to the WebSphere Application Server InfoCenter for instructions on disabling JIT compiling in a WebSphere Application Server environment at http://www.ibm.com/software/webservers/appserv/infocenter.html.
Disabling JIT compiling can affect the performance of the JVM significantly especially during periods of high load.
4XESTACKTRACE at com.ibm.ws.util.BoundedBuffer.
take(BoundedBuffer.java(Inlined Compiled Code))
4XESTACKTRACE at com.ibm.ws.util.ThreadPool.
getTask(ThreadPool.java(Compiled Code))
Example 2 – You can see the Compiled Code and Inlined Compiled Code references where the line number in a typical stack trace is observed. Threads have states, as shown in Example 1. Most threads will indicate either an R or a CW state. The following is a list of states and a brief description of each:
- R – A Runnable Thread that can run or is running. On platforms such as AIX and Linux, you can see which thread is actually running via the XHPI - Current Thread Details.
- CW – A Conditional Wait Thread is waiting on a condition variable, typically a certain condition to occur. For example, a thread may be waiting for a resource to become available.
- MW – A Monitor Wait Thread is waiting on a monitor lock.
- S – A Suspended Thread is suspended.
Looking into Monitors – LK Dump Routine
The following table shows how the LK Dump routine is broken down into five subsections:
- Monitor Pool Info provides basic monitor information, such as, the current total number of monitors, and so on.
- Monitor Pool Dump lists the monitors that exist in the JVM at the time of the Java dump along with the threads waiting for that particular monitor and the owner of each monitor. Monitors can be owned or not owned as indicated in the file. You can use this section to identify any problems, such as, too many threads waiting on a monitor.
- JVM System Monitor Dump is similar to the monitor pool dump except that instead of listing any monitor, it lists all of the system monitors on the JVM for which the Java dump was captured.
- Java Object Monitor Dump is the same as the monitor pool dump except that additional JVM internal information is provided.
- Thread Identifiers provide an association between the XM dump routine and the monitor pool dump. You can use this section as a pointer to the location of the problem identified in the monitor pool dump.
Looking into Classloaders – CL Dump Routine
The CL dump routine shows the list of class loaders and their corresponding classes. The respective address for each class and class loader is also shown in this section of the Java dump. The type for each class loader is shown to the left of the name of the class loader indicating if the class loader is primordial, extension, shareable, middleware, system, or trusted.
WebSphere Application Server Java Dumps
There are several types of threads that appear when you generate a Java dump in a WebSphere Application Server. The following list contains the types of threads that reside in most WebSphere Application Server Java dumps:
- Server Socket threads – Server Socket threads listen for the incoming request in the Web container component of WebSphere Application Server. When the threads get a request they delegate the work to the Servlet Engine Transport thread. For example, the Server Socket thread listens to incoming Web requests (port 9080, by default) and delegates the request to the Servlet Engine Transport threads. When there’s a backlog of connections, the Server Socket threads operate like the Apache Web server, which means that there’ll be a maximum of 511 backlog connections supported. When there are more than 511 clients in addition to the Servlet Engine Transport thread maximum, the next client will be refused service.
3XMTHREADINFO “ServerSocket[addr=0.0.0.0/
0.0.0.0,port=0,localport=9085]” (TID:0x10628F00,
sys_thread_t:0x4D87A80, state:R, native ID:0x9C4)
prio=5
4XESTACKTRACE at java.net.PlainSocketImpl.
socketAccept(Native Method)
4XESTACKTRACE at java.net.PlainSocketImpl.
accept(PlainSocketImpl.java(Compiled Code))
4XESTACKTRACE at java.net.ServerSocket.
implAccept(ServerSocket.java(Compiled Code))
4XESTACKTRACE at java.net.ServerSocket.
accept(ServerSocket.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.http.HttpTransport.
run(HttpTransport.java:235)
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)Example 3 – This example shows the Server Socket thread listening to port 9085 on an IBM V1.4.1 JVM and accepting a connection.
- Servlet Engine Transport threads – WebSphere Application Server maintains the Servlet Engine Transport threads in a thread pool. These threads execute the service methods of the servlets. Depending on the settings for the thread pool and the number of new incoming requests, you can create transport threads or use ones from the thread pool. The number of transport threads occupied by requests coincides with the number of unique clients connected to the Web container. If you enable the KeepAlive parameter, the connection persists as long as you specify through this parameter and the Servlet Engine Transport thread will be reserved for a particular user during this time. In other words, a blocking Input/Output (I/O) operation is performed in the Servlet Engine Transport thread.
3XMTHREADINFO “Servlet.Engine.Transports : 0”
(TID:0x10631CC8, sys_thread_t:0x4E397A8, state:R,
native ID:0xEF4) prio=5
4XESTACKTRACE at java.net.SocketInputStream.
socketRead0(Native Method)
4XESTACKTRACE at java.net.SocketInputStream.
read(SocketInputStream.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.io.Stream.read(Stream.
java:17)
4XESTACKTRACE at com.ibm.ws.io.ReadStream.
readBuffer(ReadStream.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.io.ReadStream.
read(ReadStream.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.http.HttpRequest.
readRequestLine(HttpRequest.java(Compiled Code))
4XESTACKTRACE at com.ibm.ws.http.HttpRequest.
readRequest(HttpRequest.java:302)
4XESTACKTRACE at com.ibm.ws.http.HttpConnection.
readAndHandleRequest(HttpConnection.java:596)
4XESTACKTRACE at com.ibm.ws.http.HttpConnection.
run(HttpConnection.java:443)
4XESTACKTRACE at com.ibm.ws.util.ThreadPool$Worker.
run(ThreadPool.java(Compiled Code))
Example 4 – This example represents the Servlet Engine Transport thread reading the request from the socket provided by the server socket thread.
- Object Request Broker (ORB) threads – The ORB threads do the Internet Inter-ORB Protocol (IIOP) communication and are responsible for handling Enterprise Bean communication. The ORB thread uses the Java Naming Interface (JNI) to find objects to which it must bind and connect. ORB threads have socket listeners that you’ll see in a Java dump. To understand how the ORB thread works, see the IBM JVM Diagnostics Guides in the references section of this document.
3XMTHREADINFO “ORB.thread.pool:14118” (TID:0x10B1B7B0,
sys_thread_t:0x45F9BCE0, state:CW, native
ID:0x1624) prio=5
4XESTACKTRACE at java.lang.String.valueOf(String.
java(Compiled Code))
4XESTACKTRACE at com.trade.eps...getNextInBatch
(TopCrateFactory.java:99)
4XESTACKTRACE at com.trade.eps...getNextInBatch
(TopSequenceGenerator.java:259)
4XESTACKTRACE at com.trade.eps...getNextInSequence
(TopLinkGenerator.java:220)
4XESTACKTRACE at com.trade.eps...setKeyAttributes
(TopTxnnManager.java:1138)
...
4XESTACKTRACE at com.ibm.CORBA.iiop.ORB.process
(ORB.java(Compiled Code))
4XESTACKTRACE at com.ibm.CORBA.iiop.OrbWorker.
run(OrbWorker.java(Compiled Code))
4XESTACKTRACE at com.ibm.ejs.oa.pool.
ThreadPool$PooledWorker.run(ThreadPool.java
(Compiled Code))
4XESTACKTRACE at com.ibm.ws.util.CachedThread.
run(ThreadPool.java(Compiled Code))Example 5 – This example represents the server ORB thread executing an Enterprise Bean.
- Java Message Service (JMS) threads – JMS threads use additional threads to do work for message-driven beans. Calls to message-driven beans are asynchronous with respect to an entire application. However, the actual message-driven bean onMessage() method invocation is a synchronous operation that you can view in a Java dump. A message-driven bean onMessage() method call will wait (block) on any I/O that’s being performed, typically another Enterprise Bean call.
- Miscellaneous threads – WebSphere Application Server contains many other threads that do internal tasks such as alarm threads, thread pool management, and so on.
Initiating Java Dumps on WebSphere Application Server 5.x
You can initiate the WebSphere Application Server Java dumps using the wsadmin tool just follow these steps:
- Start the wsadmin tool in the WebSphere/AppServer/bin/wsadmin.sh (or .bat on Windows).
- Get a handle on the JVM process for which you want to initiate a dump. Typically this is the WebSphere Application Server process set by default to server1.
- Invoke the dumpThreads method using the AdminControl object. The following example demonstrates the code used to do this for a default server1 process once you invoke the wsadmin tool. See the WebSphere Application Server InfoCenter for more information.
wsadmin>set jvm [$AdminControl completeObjectName
type=JVM,process=server1,*]
WebSphere:platform=common,cell=merengue51BaseNode,
version=5.0.1,name=JVM,mbeanIdentifier=JVM,type=JVM,n
ode=merengue51BaseNode,process=server1
wsadmin>$AdminControl invoke $jvm dumpThreads
Example 6 – This example demonstrates invoking a Java dump using the wsadmin tool. By default these files are generated in the WebSphere/AppServer/ directory. However, you can specify the location using an environment variable supplied to the JDK. For more details on this environment variable, invoking Java dumps using operating system-specific signals, and other options for invoking Java dumps, see the IBM JVM Diagnostics Guides in the reference section of this article.
Types of Problems Detectable by a Java Dump
The following list includes problem indicators in a Java dump. For many of these problems you can use ThreadAnalyzer to detect and determine the cause:
- Monitors that have a long or longer than usual list of threads waiting for them. They indicate that your application has a larger number of threads waiting on the monitor than it would normally have. If this situation occurs, a bottleneck may exist due to improper monitor use. There’s no set number of threads that indicates this problem because it is dependent on the application, such as the way it handles synchronization. To find and fix this problem, view the monitors listed in the LK subcomponent dump routine (see Looking into Monitors – The LK Dump Routine to understand the LK dump routine). Look for the monitors with an unusually high number of threads waiting on them. For example, if most of the monitors have one or two threads waiting for them compared to those that have six or eight threads waiting on them, it may indicate a performance bottleneck at those resources. If the resource never becomes available, the application may hang. If that happens, follow these steps:
- Determine which thread owns the monitor that’s creating the bottleneck by looking at the LK dump routine and singling out the monitor that seems to have the longest (abnormal) list of threads waiting on it. You can identify by the number of threads that follow the text “Waiting to be notified.”
- View the XM subcomponent dump routine for the details of the thread that owns the monitor (the one with the large number of waiting threads) by comparing the thread identification (provided in the LK dump routine) with the XM dump routine.
- Determine the state of the thread and do one of the following:
- If the thread is in the runnable state, determine which part of the code is executing by lookingat the stack trace. Many times this section of the code is the bottleneck.
- If that thread is in a conditional wait state (CW) or monitor wait state (MW), go to the LK subcom ponent dump routine to see if the thread is waiting for another resource. Repeat the process to find the potential source of the bottleneck.
- Saturated thread pools – This is a thread pool where the number of executing threads is equal to or greater than the maximum number of threads allocated to it. If a particular thread pool isn’t growable (the isGrowable flag isn’t enabled), a saturated thread pool will show the maximum allocated pool size is already allocated. If the thread pool is growable, then the thread pool is saturated when the server doesn’t perform well due to an overly grown thread pool. When your thread pools are saturated, you can see what is bappening with each of the threads in the thread pool. To figure out which thread pool the thread represents, look at the stack trace to understand the underlying code that calls thread pool workers or the name of the threads in a particular thread pool. For problem determination, it’s important to understand the cause of the thread pool saturation. In many cases, it’s because of a misuse of synchronization practices or poorly configured thread pool size. If you have a synchronization problem, you’ll see that the threads in the thread pool are waiting on a shared resource. Otherwise you may have a poorly configured thread pool size and you should try tuning it.
- Circular dependencies among threads, such as deadlocks. For each thread, determine the resources that they own and are waiting on based on the information provided in the XM and LK subcomponent dump routines. This information includes the correlation of the TID and sys_thread_t of the thread and the name of the monitor. Determining circular dependencies can be complicated. However, some JDKs and tools such as ThreadAnalyzer do the computations for you automatically.
- Threads running indefinitely that shouldn’t be running indefinitely. These threads can be seen in a runnable state after taking multiple Java dumps separated by a three-five minute interval. If you look at the stack trace in the XM subcomponent dump routine and the threads are still executing the same part of the code (compare the thread, sys_thread_t, and native thread identifications to all captured Java dumps), it’s possible that these threads will run indefinitely. If other threads are waiting on resources (monitors) that the indefinitely running threads own, then the other threads will hang indefinitely.
- Threads waiting indefinitely. These threads can be seen waiting for the same resource in multiple Java dumps separated by a three-five minute interval. In this case, identify the resource that the threads are waiting for by comparing the thread name, thread identification, sys_thread_t, and native thread identification for the thread with the captured Java dumps. Typically this is caused by the thread waiting for a monitor that you can see in all the captured Java dumps. If the thread is waiting for a monitor, the state of the thread is either CW or MW and you can find the thread on the LK component dump routine as “Waiting to be notified” or “owned.”
Classloader Problems
If the application experiences problems or errors related to Java classes missing at runtime, determine which class file is missing by looking at the classes that are loaded by the class loader and listed under the CL subcomponent dump routine. If you can’t find the class that’s in the CL subcomponent dump routine, make sure you can find the class in the file system under the appropriate structure. Otherwise, make sure that the class file is being loaded by the proper class loader. For further information, go to the WebSphere Application Server InfoCenter or the references listed below.
ThreadAnalyzer
ThreadAnalyzer is a problem determination tool for WebSphere Application Server thread performance and deadlock detection. You can use it to obtain a Java dump from an application server and use its analysis features for problem determination. For performance problem determination, ThreadAnalyzer provides a top-of-the-stack (TOS) analysis that counts all of the methods at the top of the stack for the listed threads. It then sorts them by ‘weight’ (automatically computed by ThreadAnalyzer), and lists the possible performance bottlenecks in your application. ThreadAnalyzer does automatic deadlock detection while it analyzes the Java dump.
Downloading ThreadAnalyzer
To download ThreadAnalyzer:
- Go to http://www.ibm.com/developerworks/websphere/downloads/thread_analyzer.html
- Follow the links to the login. If you don’t have an account, click the “I have not registered before” link.
- Follow the rest of the links to download ThreadAnalyzer.
To install ThreadAnalyzer, extract the contents of the zip file to anywhere in your file system. To start ThreadAnalyzer make sure you have WebSphere Application Server V4 or V5 installed and then start the tagui.cmd or tagui.sh file located in the <ThreadAnalyzerInstall>/bin directory. For more information on ThreadAnalyzer, see the product documentation provided in the zip file on installation.
References
- IBM JVM Diagnostics Guides - http://www.ibm.com/developerworks/java/jdk/diagnosis/
- JDK 1.3.1 and JDK 1.4.1 for WebSphere Application Server V5.0 and V5.1
- WebSphere Application Server Infocenters www.ibm.com/software/webservers/appserv/infocenter.html
Many thanks for the contributions provided by Steve Eaton, Daniel Julin, and Arvind Srinivasan.