A: We've recently added some new dtrace(1M) options to make this exact task easier. I'll describe them below. They will appear in future SX builds based on s10_63 or later.

    BASE    LIMIT     SIZE NAME
   10000   10e000    fe000 a.out
ff3c0000 ff3ea000    2a000 ld.so.1

You are correct that it is "too early" in the lifetime of the process to instrument libc. When you use the DTrace pid provider, we grab the process you are instrumenting and inspect its symbol tables to determine where to place the appropriate instrumentation. In the above mdb -p output, you can see that you have stopped the process very early in its userland lifetime, where it is still in the dynamic linker processing dependencies. So libc has not even been mapped into the address space yet. As you discovered, you need to wait until just before main() is called to be sure that all the libraries are mapped in. We've added some new options to make this easier.

In the newest version of DTrace which will be part of the next Solaris Express and the final version of S10, we've added dtrace(1M) options -c and -p to create and run a command and grab a particular process. These will form the basis of further enhancements for userland tracing as well. The -c option in particular lets you run a command up until the point where the dynamic linker has done its work, stop it, and then activate instrumentation exactly as you were doing by hand. We've also added a new $target macro variable, which expands to the process ID of the thing you created or grabbed with -p, to make things easier. So you'll be able to say, trace every function call in libc in the date(1) command like this:

# dtrace -c date -n 'pid$target:libc.so::entry{trace(pid);}'
dtrace: description 'pid$target:libc.so::entry' matched 2464 probes
Tue Aug 10 18:08:22 PDT 2004
dtrace: pid 930373 has exited


CPU     ID                    FUNCTION:NAME
  0  36039                  libc_init:entry    930373
  0  34168                     atexit:entry    930373
  0  35697                    lmalloc:entry    930373
  0  35696               getbucketnum:entry    930373
  0  35695         initial_allocation:entry    930373
  0  36220                       mmap:entry    930373
  0  36302               __getcontext:entry    930373
  0  35697                    lmalloc:entry    930373
  0  35696               getbucketnum:entry    930373
  0  36206                  getrlimit:entry    930373
  0  36203                     getpid:entry    930373
  0  35697                    lmalloc:entry    930373
  0  35696               getbucketnum:entry    930373
...
  0  33969                       exit:entry    930373
...
  0  36364                      _fini:entry    930373
  0  36040                  libc_fini:entry    930373
...
  0  35953               mutex_unlock:entry    930373


#

Stay tuned to Solaris Express for this version.

Back to top

A: DTrace only runs on Solaris 10, available now through the Solaris Express download program. But you can use DTrace to find a problem in your app by running the app on Solaris 10, improve your app, and then put your app back into production on Solaris 8 and reap the benefits.

Back to top

A: Yes. We are currently working on a SunEd course which should be available by the time Solaris 10 becomes generally available. We also intend to ultimately provide advanced, domain-specific courses -- but those courses will be developed after we get some experience with the initial DTrace course.

Thank you for creating DTrace!

We're glad that it's useful to you!

Back to top

A: Yes, it would. It's certainly possible today to use DTrace to understand an application-specific problem, but the most useful scripts almost always require encoding some implementation knowledge.

We are addressing this by adding extensions to DTrace that will allow applications to export probes with semantic stability. So just as we have the stable "io", "proc" and "sched" providers today, we hope to one day have (for example) a stable "apache" provider that makes probes available that have semantic relevance to those administering Apache. (It will ultimately be up to the Apache developers to define and maintain these probes, so we cannot commit to the provider itself -- only to the technology that it will be built upon.) And the sample scripts of these providers should fall out of the documentation of them...

Back to top

A: Right now we don't have demangling built in -- we're working on adding that feature. You can demangle things by hand by using the "dem" utility that comes with your Sun C++ compiler or by using the "gc++filt" utility that comes with GCC.

Back to top

A: Right now the major piece of Java observability is the the DTrace jstack() action (alias for ustack() if it's a Java process) which will capture a Java source-level stack trace if the process is a "Tiger" JVM, which is the latest one. You can also use the normal DTrace probes to relate Java activity to underlying system activity such as i/o and scheduling etc.

Back to top

A: Currently DTrace can't be used from within a local (non-global) zone. We're working to make this possible, but it won't be available in the first release of the Solaris 10 OS (we may later add that in an update release).

You can, however, use DTrace in the global zone to trace activity in other zones. The DTrace 'zonename' variable also lets you predicate, aggregate, or whatever on the zone causing the activity.

Back to top

A: You don't have to make any modifications to your application or libraries in order to trace them using DTrace's pid provider -- you don't even have to recompile your binaries. The only adjustment you might want to make is to ensure you don't strip the symbol table. The pid provider lets you trace every function's entry and return as well as any instruction at any function offset, as long as there's a symbol table entry for the function.

If there are points in your code where you'd like explicit probes or if you'd like to have multiple disparate points in your code activate a single, stable probe, you will be able to use User Land Statically Defined (USDT) tracing to embed probes. That functionality is not available in the current Solaris Express release, but it is scheduled for the final release of the Solaris 10 OS.

In terms of kernel modules, you can trace function entry and return for most functions in all kernel modules (there are some kernel functions that may be unsafe to instrument). Again, you don't need to recompile the binaries.

Back to top