Tom Gissel @ singlethreaded.org

In response to my last post a colleague said:

Good post…..how did you create the HeapDump, and how did you view it?

Good questions.

A HeapDump is simply a dump of the live objects in a JVM’s heap at a given point in time. HeapDumps are usually used to debug memory leaks or to find the big memory consumers within a JVM. While most JVMs provide the ability to create HeapDumps the generation techniques vary from JRE to JRE.

IBM’s JRE has many options to deal with the generation and format of it’s HeapDumps, for all the details take a look at the the diagnostic guide. By default a HeapDump is only produced when an java.lang.OutOfMemoryError is hit. This can be seen by executing “java -Xdump:what”

gissel@uw-t60p:~/Workings/leak> java -Xdump:what

Registered dump agents
———————-
dumpFn=doSystemDump
events=gpf+abort
filter=
label=/home/gissel/Workings/leak/
core.%Y%m%d.%H%M%S.%pid.dmp
range=1..0
priority=999
request=serial
opts=
———————-
dumpFn=doSnapDump
events=gpf+abort
filter=
label=/home/gissel/Workings/leak/
Snap%seq.%Y%m%d.%H%M%S.%pid.trc
range=1..0
priority=500
request=serial
opts=
———————-
dumpFn=doSnapDump
events=uncaught
filter=java/lang/OutOfMemoryError
label=/home/gissel/Workings/leak/
Snap%seq.%Y%m%d.%H%M%S.%pid.trc
range=1..4
priority=500
request=serial
opts=
———————-
dumpFn=doHeapDump
events=uncaught
filter=java/lang/OutOfMemoryError
label=/home/gissel/Workings/leak/
heapdump.%Y%m%d.%H%M%S.%pid.phd
range=1..4
priority=40
request=exclusive+prepwalk
opts=PHD
———————-
dumpFn=doJavaDump
events=gpf+user+abort
filter=
label=/home/gissel/Workings/leak/
javacore.%Y%m%d.%H%M%S.%pid.txt
range=1..0
priority=10
request=exclusive
opts=
———————-
dumpFn=doJavaDump
events=uncaught
filter=java/lang/OutOfMemoryError
label=/home/gissel/Workings/leak/
javacore.%Y%m%d.%H%M%S.%pid.txt
range=1..4
priority=10
request=exclusive
opts=
———————-

This is a good setting when running in a production environment, but not the best for debugging. When debugging I usually like to take several HeapDumps during a test, as the heap grows, to see what what type of object is growing. This is best done by sending a SIGQUIT signal to the JVM process. To enable the creation of HeapDumps via the SIGQUIT signal set the JVM option “-Xdump:heap”. The total effect of this option can be seen by executing “java -Xdump:heap”

gissel@uw-t60p:~/zero-1.0.0.P20070712-1334/apps/employee.demo> java -Xdump:heap:?
JVMDUMP000E Dump Option unrecognised: -Xdump:heap:…?

Capture raw heap image:

-Xdump:heap[:defaults][:<option>=<value>, ...]

Dump options:

events=<name> Trigger dump on named events
[+<name>...] (see -Xdump:events)

filter=[*]<name>[*] Filter on class (for load,throw,catch,uncaught)
#<n>..<m> Filter on exit code (for vmstop)

file=<label> Output file
range=<n>..<m> Limit dumps
priority=<n> Highest first
request=<name> Request additional VM actions
[+<name>...] (see -Xdump:request)

opts=PHD|CLASSIC

Default -Xdump:heap settings:

events=gpf+user
filter=
file=/home/gissel/zero-1.0.0.P20070712-1334/apps /employee.demo/
heapdump.%Y%m%d.%H%M%S.%pid.phd
range=1..0
priority=40
request=exclusive+prepwalk
opts=PHD

Could not create the Java virtual machine.

An interesting option is “opts”. As seen above, the default “opts” value is PHD. PHD is actually the format of the HeapDump the other options are CLASSIC(text file), and PHD+CLASSIC (produces 2 files for each requested HeapDump, one in PHD and the other in TXT format). The HeapDump file format is important because some of the tools can only read either PHD or TXT format.

The tool I use to view HeapDumps varies depending upon which tool is most up-to-date . Currently, the first tool I use to get a general feel for what is in the heap is HeapAnalyzer. I also like HeapRoots.

Try the following test. Download the attached code, extract the archive, execute ./runscript.groovy, and monitor the resident memory used by the script. In my tests the process footprint steadily grows until the script exits. This is type of behavior is usually indicative of a memory leak; however, upon inspecting the script it doesn’t look like it should be leaking. What is going on?

A HeapDump taken during the script’s execution shows that one of org.codehaus.groovy.runtime.ReferenceMap’s HashMaps has grown to an enormous size containing thousands of entries, each key being a reference to a groovy.lang.MetaClassImpl instance. It seems that every execution of shell.evaluate creates, among other things, a new MetaClassImpl object which is subsequently stored within the ReferenceMap. The longer the script run the larger ReferenceMap grows.

These seems like a proto-typical memory leak. MetatClass objects define behavior for specific java/groovy class. Ironically most of the classes for which the MetaClass objects exist are no longer in memory thus are pure garbage. Yet, the MetaClass objects remain resident due to their existence in the ReferenceMap. A closer look at the HeapDump shows that ReferenceMap’s HashMap is a WeakHashMap whose keys are soft references. This means that the HashMap entries should eventually be collected. When the entries are collected is determined by the GC policy of the VM. By default the IBM JDK collects soft references every 32 garbage collection cycles. As it turns out 32 cycles can take a very long time. Luckily the default soft reference policy can be overridden by supplying the -Xsoftrefthreshold parameter. Voila! Setting -Xsoftrefthreshold1 eliminates the exponential process growth producing a maximum resident memory size to 39MB.

I mentioned that I don’t believe that Java has any true scripting languages because none of its languages are useful for System Administration. One of specific flaws that all Java scripting languages share is general sluggishness. This begs the question: How fast is fast enough?

To answer the question “How fast is fast enough” the question must be put into some context. In this discussion the context is system administration. Complex language tests like mandelbrot are not suitable when speaking in terms of system administration because by their very nature shell scripts are supposed to start-up, perform a simple task, and shutdown. A more appropriate test is measuring how quickly the language can perform an extremely simple task like adding 1+2. For this type of test my threshold for “fast enough” is visually noticing a lag from the time I hit enter on the keyboard to the time the script returns. For me, this equates to about 60ms. By this measure none of the Java scripting languages are fast enough.

Taking the question from the subjective to the objective it is important to understand just how far Java’s scripting languages are from its contemporaries. So, I benchmarked start-up time for the following languages on my Thinkpad T60p with an Intel T2600 processor: BASH, python, perl, jython, jruby, and groovy. The results are listed below.

Language	Start-up Time (ms)
BASH	8
python	27
perl	7
ruby	10
jython	900
jruby	1900
groovy	1200

The number speak for themselves, Java’s scripting languages are at least an order of magnitude slower than its non Java counterpart.