BigAdmin Feature Article: Observability Using Java Platform, Standard Edition 6, and Solaris OS

BigAdmin System Administration Portal

Feature Article

Submit Content | Check BigAdmin Bucks

Observability Using Java Platform, Standard Edition 6, and Solaris OS

A. Sundararajan and Jim Holmlund, May 2006 (Updated December 2008)

Java Platform, Standard Edition (Java SE) 6 is the latest Java SE release under development. Java SE 6 source and binary snapshot downloads have been available on a regular basis from the JDK 6 project on Java.net.

The Java SE 6 platform offers various observability tools. Many of these tools can be used on production systems against running programs. Few of these tools are for processes that are hung or core dumps (port-mortem analysis). Along the way, we will see the effect of these observability tools on observed processes.

Observability Tools in the Java SE 6 Platform Using DTrace

Many observability improvements are found in Java SE 6 software. While most of these are applicable for all platforms, a few of these enhancements are specific to the Solaris Operating System, in particular Solaris 10 and higher releases. In Java 2 Platform, Standard Edition 5.0 (J2SE 5.0 platform), a new Dynamic Tracing (DTrace) action called jstack was added. jstack, as you may know already, prints mixed-mode stack traces (frames in both Java and native C/C++ languages are shown). The following D script prints a mixed-mode stack trace whenever the pollsys system call is made from a given Java process:

#!/usr/sbin/dtrace -s
syscall::pollsys:entry
/ pid == $1 /  {
    /* print at most 50 frames */
    jstack(50);
}

Here is the output of running the above script against a Java process:

libc.so.1`__pollsys+0xa
libc.so.1`poll+0x52
libjvm.so`int os_sleep(long long,int)+0xb4
libjvm.so`int os::sleep(Thread*,long long,int)+0x1ce
libjvm.so`JVM_Sleep+0x1bc
java/lang/Thread.sleep
dtest.method3
dtest.method2
dtest.method1
dtest.main
[... more output deleted for brevity ...]

The jstack action is a very useful starting point in Java observability on the Solaris 10 OS and above. But, Java SE 6 technology goes beyond that to add many DTrace probes for Java Virtual Machine (JVM) observability and Java application observability. The Java SE 6 release adds two JVM-specific DTrace providers, namely hotspot and hotspot_jni.

The `hotspot` Provider

The hotspot provider adds many probes, which may be roughly classified as follows:

VM life cycle probes: VM start, shutdown events
Thread life cycle probes: thread start, stop, and so on
Class-loading probes: a Java class load or unload
Garbage collection probes: GC start, end
Method compilation probes: Java bytecode-to-native code compilation ("hotspot compilation")
Monitor probes: Java monitor contended entry, notify, notifyAll, and so on
Application probes: Java object alloc, Java method entry/return, and so on

Exact details of these probes can be found in Keith McGuigan's blog. Following are a few examples of the use of these probes.

Printing the names of Java threads as they are started:

hotspot$1:::thread-start {
   self->ptr = (char*) copyin(arg0, arg1+1);
   self->ptr[arg1] = '\0';
   self->threadname = (string) self->ptr;
   printf(“Thread %s started\n”, self->threadname);
}

-verbose:class equivalent in D script:

You may have used the -verbose:class JVM option. This option causes the JVM to print a trace message whenever a Java class is loaded or unloaded. Here is how to get the same output using DTrace:

/* print the name of each class as it is loaded */
hotspot$1:::class-loaded {
    self->str_ptr = (char*) copyin(arg0, arg1+1);
    self->str_ptr[arg1] = '\0';
    self->name = (string) self->str_ptr;
    printf(“class %s loaded\n”, self->name);
}

printStackTrace on exception (but mixed mode):

The Throwable.printStackTrace() method prints a stack trace containing Java frames. This D script will print all frames -- for Java code, Java Native Interface (JNI) code, C/C++ code, and OS C/C++ code -- whenever an exception is thrown.

hotspot$1:::method-entry {
    self->ptr = (char*)copyin(arg1, arg2+1);
    self->ptr[arg2] = '\0';
    self->classname = (string)self->ptr;
    self->ptr = (char*)copyin(arg3, arg4+1);
    self->ptr[arg4] = '\0';
    self->methodname = (string)self->ptr;
}

hotspot$1:::method-entry 
/ self->classname == "java/lang/Throwable" && self->methodname == "<init>" / {
   jstack();
}

All exceptions and errors in the Java code are directly or indirectly derived from java.lang.Throwable, so all exception constructors will eventually call the constructor of java.lang.Throwable. This D script sets up a method-entry probe with a filter for that constructor (note that <init> is the internal name for constructor methods). The jstack() action will print the mixed-mode stack trace when this constructor is called.

Java Heap Histogram

We can use the object-alloc probe to construct a Java heap histogram as shown here:

hotspot$1:::object-alloc {
    self->str_ptr = (char*) copyin(arg1, arg2+1);
    self->str_ptr[arg2] = '\0';
    self->classname = (string) self->str_ptr;
    @allocs_count[self->classname] = count();
    @allocs_size[self->classname] = sum(arg3);
}

The `hotspot_jni` Provider

JNI allows Java programs to interact with C/C++ code. If you want to observe Java code-to-native code interactions, you can use the hotspot_jni provider. For every JNI function entry/exit, this provider exposes a probe.

Effect of Using DTrace With Java Programs

DTrace is designed as a production-mode observability system offering "zero impact when not used." When probes are not enabled, the observed system is not affected.

For the most part, the JVM probes exposed by hotspot and hotspot_jni providers are very lightweight and so can be used on production machines. However, a few probes exposed by the hotspot provider require the JVM to be started with a special command-line option called -XX:+ExtendedDTraceProbes. These are the Java method-entry/method-return, object-alloc, and Java monitor probes. These probes require changes in the hotspot bytecode interpreter and hotspot compiler (bytecode-to-machine code compiler). These probes are comparatively costly even when not enabled.

Observability Tools in the Java SE 6 Platform

In addition to the integration of DTrace with Java technology, the Java SE 6 release contains many other observability tools. A summary of these tools follows, with links to references that offer more details.

VisualVM

Since Version 6 update 7, the Java Development Kit has come with a new troubleshooting tool called Java VisualVM. This tool provides a visual interface for viewing detailed information about Java applications while they are running on a Java Virtual Machine (JVM), and for troubleshooting and profiling these applications.

Java VisualVM can be used by Java application developers, system administrators, and system integrators to troubleshoot applications and to monitor and improve the applications' performance. Java VisualVM can be used to generate and analyze heap dumps, track down memory leaks, browse the platform's MBeans and perform operations on those MBeans, perform and monitor garbage collection, and perform lightweight memory and CPU profiling. Furthermore, Java VisualVM can be extended to add new functionality by creating and posting plug-ins to the tool's built-in update center.

For more information see Java VisualVM on java.sun.com. Java VisualVM can also be downloaded as a standalone tool from java.net/ (see the VisualVM Open Source project).

JConsole

JConsole uses the extensive Java Management Extension (JMX) instrumentation of the JVM to provide information about the performance and resource consumption of applications running on the Java platform. (See Using jconsole.)

In J2SE 5.0 software, you need to start the application to be monitored with the -Dcom.sun.management.jmxremote option. Note: With Java SE 6 software, this requirement goes away. No special command-line option is needed when starting the application.

Use in Production Systems

JConsole starts a JMX agent inside the JVM of the observed Java program. A non-zero impact results from the running of an additional agent -- but the impact is minimal.

Also, despite the fact that JConsole is useful in monitoring local applications for development and prototyping, this is not recommended for production environments. The reason is that JConsole itself consumes significant system resources. Remote monitoring is recommended to isolate the JConsole application from the platform being monitored. So, for production systems, it is better to use JConsole in remote mode. Security options are available for secure remote monitoring.

jps

jps is equivalent to the Solaris proc tool ps. For more information, see jps - Java Virtual Machine Process Status Tool.

Unlike pgrep java or ps -ef | grep java, jps does not use the application name to find JVM instances. Thus, it finds all Java applications, even the ones that don't use the java executable (for example, custom launchers). Also, jps finds only the Java processes of the current user, rather than all the processes in the system.

jstat

jstat displays performance statistics for an instrumented Java HotSpot virtual machine. (See jstat - Java Virtual Machine Statistics Monitoring Tool.) More details on performance counters are found here: Code sample - jvmstat 3.0.

jstatd

jstatd is a Java Remote Method Invocation (RMI) server application that monitors the creation and termination of instrumented Java HotSpot virtual machines and provides an interface to allow remote monitoring tools to attach to JVMs running on the local host. (Refer to jstatd - Virtual Machine jstat Daemon.)

Use in Production Systems

jps and other jvmstat utilities use very lightweight observation mechanisms. A small piece of shared memory is allocated by the JVM, and performance counters are allocated from the same. The subsystems of the JVM update the performance counters based on interesting events. The client tools just read from the shared memory segment asynchronously. So, overall, the effect of monitoring with jvmstat is minimal.

Tools for Postmortem Observability in the Java SE 6 Platform

Java SE 6 technology supports postmortem observability tools that obtain information from hung Java processes or Java core dumps. These tools (except for jhat) use the Solaris libproc library to attach and read from the observed program. During the period of observation, the target program is suspended. These tools are expected to be used when Java processes are hung or when a core dump from a Java process has occurred. Whenever possible, consider using gcore to take a snapshot core dump of the system and analyze the core dump "offline" using any of the following tools.

jinfo

jinfo prints Java configuration information for a given Java process or core file or a remote debug server. Configuration information includes Java system properties and JVM command-line flags. (For more information, see jinfo - Configuration Info.)

jmap

jmap: If this tool is run without any options (other than pid or core), then it displays information similar to that of the Solaris pmap tool. This tool supports several other options for Java heap observability. (Refer to jmap - Memory Map.)

In the Java SE 6 platform, a new -dump option has been added. This causes jmap to dump the Java heap into a file, which can then be examined using the new jhat command (see below).

The jmap -dump option does not use Solaris libproc for live processes -- instead, it runs a small piece of code in the running JVM, which dumps the heap. Since this heap dumping code runs inside the JVM, it is relatively fast. The effect of heap dumping is roughly the same as doing a "full GC" (garbage collection for the entire heap) plus writing the contents of the heap to the file. One other possible route for a heap dump is to use gcore to take the core dump and run jmap -dump against the core dump in "offline" mode.

jstack

jstack is equivalent to the Solaris pstack tool. jstack prints stack traces for all Java threads, optionally including native frames. (See jstack - Stack Trace.) Information about locks and deadlocks can also be printed, as shown in java.util.concurrent locks.

jsadebugd

jsadebugd attaches to a Java process or core file and acts as a debug server, as shown in jsadebugd - Serviceability Agent Debug Daemon. Remote clients such as jstack, jmap, and jinfo can attach to the server using Java RMI.

jhat

jhat is a Java heap dump browser (see jhat - Java Heap Analysis Tool). This tool parses a Java heap dump file (for example, one produced by jmap -dump, see above). jhat starts a web server that allows objects in the heap to be examined in a web browser. This tool is not meant to be used on production systems and is meant for "offline" analysis of heap dumps. The jhat tool is platform independent in the sense that it can be used to view heap dumps produced on any platform. For example, it is possible to view a heap dump produced on the Solaris OS by using jhat on Linux.

jvisualvm

The Java VisualVM tools distributed with the JDK since JDK 6 update 7 can also be used to analyze heap dumps, thread dumps, and core files. See in particular Working with Core Dumps in the Java VisualVM guide.

References

Blogs on Solaris and Java Platform Observability

Articles and Guides

Using JConsole to Monitor Applications
The Solaris Dynamic Tracing Guide
Java 2 Platform, Standard Edition 5.0, Java Troubleshooting and Diagnostic Guide (This information is scheduled to be updated for the Java SE 6 platform in future.)
Core Java Technology Features in Java SE 6

Updates

December 2008: Added sections on VisualVM under Observability Tools in the Java SE 6 Platform and also Tools for Postmortem Observability in the Java SE 6 Platform.