Print-friendly Version

Observability Using Java 2 Platform, Standard Edition 5.0, and the Solaris OS

A. Sundararajan and Jim Holmlund, May 2006

Java 2 Platform, Standard Edition 5.0 (J2SE 5.0), is the latest FCS release available for download from Java.sun.com. The J2SE 5.0 platform comes with a number of observability tools, especially on the Solaris 10 Operating System.

Support for Java Technology Using DTrace

The Solaris 10 OS introduced Dynamic Tracing (DTrace). From J2SE 5.0 update 1 onwards, a new DTrace action called jstack has been added. jstack prints mixed-mode stack traces (frames in both Java and native C/C++ languages are shown). For example, the following D script prints mixed-mode stack traces whenever the pollsys system call is made from a given Java process:

#!/usr/sbin/dtrace -s
syscall::pollsys:entry
/ pid == $1 /  {
    /* print at most 50 frames */
    jstack(50);
}

Here is the output of running the above script against a Java process:

libc.so.1`__pollsys+0xa
libc.so.1`poll+0x52
libjvm.so`int os_sleep(long long,int)+0xb4
libjvm.so`int os::sleep(Thread*,long long,int)+0x1ce
libjvm.so`JVM_Sleep+0x1bc
java/lang/Thread.sleep
dtest.method3
dtest.method2
dtest.method1
dtest.main
[... more output deleted for brevity ...]

The jstack action is a very useful starting point in observability for Java code on the Solaris 10 OS. But what about Java technology-specific DTrace probes for better observability on the Java platform?

Using DTrace Agent for Java Virtual Machines

One DTrace agent library is called the "dvmti" agent. This can be run inside the Java Virtual Machine (JVM) to provide DTrace probes that are specific to the JVM and the Java platform. This agent is available for download from the Java.net project solaris10-dtrace-vm-agents. The dvmti agent uses the Java Virtual Machine Tool Interface (JVM TI). Specifically, the agent library utilizes the JVM TI interface to request various VM events and provide DTrace probes in the callback code for those events. To use these DTrace probes for the JVM and Java code, the Java application has to be started as shown here:

java -Xrundvmti[:options] <MainClass>

Where options are:

 	all		same as: unloads,allocs,stats,methods
	help		print help message
	unloads		track class unloads
	allocs		track object allocations and frees
	stats		generate heap stats after each GC
	methods		generate method entry exit events
	exclude=name	exclude class names

default is none of unloads, allocs, stats, or methods. 

The dvmti agent will utilize byte code instrumentation (BCI) where needed and will need a jar file called dvmti.jar, which should be located in the same directory as the shared library libdvmti.so. The JDK 5.0 release has an environment variable that can be used to include options in the startup of any Java application:

export JAVA_TOOL_OPTIONS="-Xrundvmti:all"

Effect of Using DTrace on Java Programs

DTrace is designed as a production-mode observability system offering "zero impact when not used". When probes are not enabled, the observed system is not affected. The jstack() DTrace action does not have much of an impact when used. However, a few of these dvmti-supported DTrace probes (for example, method-entry) require BCI and hence alter the target (observed) program in a significant way.

Observability Tools in the J2SE 5.0 Platform

In addition to the integration of DTrace with Java technology, J2SE 5.0 software contains many other observability tools. A summary of these tools follows, with links to references that offer more details.

JConsole

JConsole uses the extensive Java Management Extension (JMX) instrumentation of the JVM to provide information about performance and resource consumption of applications running on the Java platform. (For more information, see the article Using JConsole to Monitor Applications.) In J2SE 5.0 technology, the application to be monitored has to be started with the -Dcom.sun.management.jmxremote option.

Use in Production Systems

JConsole starts a JMX agent inside the JVM of the observed Java program. A non-zero impact results from the running of an additional agent -- but the impact is minimal.

Also, despite the fact that JConsole is useful in monitoring local applications for development and prototyping, this is not recommended for production environments. The reason is that JConsole itself consumes significant system resources. Remote monitoring is recommended to isolate the JConsole application from the platform being monitored. So, for production systems, it is better to use JConsole in remote mode. Security options are available for secure remote monitoring.

jps

jps is equivalent to the Solaris proc tool ps. For more information, see jps - Java Virtual Machine Process Status Tool.

Unlike pgrep java or ps -ef | grep java, jps does not use the application name to find JVM instances and so it finds all Java applications, even the ones that don't use the java executable (for example, custom launchers). Also, jps finds only the Java processes of the current user, rather than all the processes in the system.

jstat

jstat displays performance statistics for an instrumented Java HotSpot virtual machine, as seen in jstat - Java Virtual Machine Statistics Monitoring Tool. Code sample: jvmstat 3.0 offers more details on performance counters.

jstatd

jstatd is a Java Remote Method Invocation (RMI) server application that monitors the creation and termination of instrumented Java HotSpot virtual machines and provides an interface to allow remote monitoring tools to attach to JVMs running on the local host. For more information, see jstatd - Virtual Machine jstat Daemon.

Use in Production Systems

jps and other jvmstat utilities use very lightweight observation mechanisms. A small piece of shared memory is allocated by the JVM, and performance counters are allocated from the same. The subsystems of the JVM update the performance counters based on interesting events. The client tools just read from the shared memory segment asynchronously. So, overall the effect of monitoring using jvmstat is minimal.

Tools for Postmortem Observability in the J2SE 5.0 Platform

J2SE 5.0 software supports postmortem observability tools that obtain information from hung Java processes or Java core dumps. These tools use the Solaris libproc library to attach and read from the observed program. During the period of observation, the target program is suspended. These tools are expected to be used when Java processes are hung or when you have a core dump from a Java process. Whenever possible, consider using gcore to take a snapshot core dump of the system (see the man page gcore (1) - get core images of running processes). You can then analyze the core dump "offline" using any of the tools below.

jinfo

jinfo prints Java configuration information for a given Java process or core file or a remote debug server. (See jinfo - Configuration Info.) Configuration information includes Java system properties and JVM command-line flags.

jmap

For information on jmap, see jmap - Memory Map. If the jmap tool is run without any options (other than pid or core), then it displays information similar to that of the Solaris pmap tool. This tool supports several other options for Java heap observability.

Starting from J2SE 5.0 update release 1.5.0_05, a new -heap:format option has been added (see Bug ID 5109224, SA utilities: add heap dump option for jmap). This causes jmap to dump the Java heap into a file that can then be examined offline.

jstack

jstack is equivalent to the Solaris pstack command. jstack prints stack traces for all Java threads, optionally including native frames. Information about deadlocks, if any, is also printed. (See jstack - Stack Trace.)

jsadebugd

jsadebugd attaches to a Java process or core file and acts as a debug server. For details, see jsadebugd - Serviceability Agent Debug Daemon. Remote clients such as jstack, jmap, and jinfo can attach to the server using Java RMI.

References

Blogs on Solaris and Java Platform Observability

• Adam Leventhal
• Alan Bateman
• Eamonn McManus
• Keith McGuigan
• Kelly Ohair
• Mandy Chung
• A. Sundararajan

Articles and Guides

• Using JConsole to Monitor Applications
• The Solaris Dynamic Tracing Guide
• Java 2 Platform, Standard Edition 5.0, Trouble-Shooting and Diagnostic Guide (pdf)
• JVM TI: How VM Agents Work

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.