Back to newsletter 075 contents

http://www.fasterj.com/articles/hotpatch1.shtml
Hotpatching A Java 6 Application (Page last updated February 2007, Added 2007-02-27, Author Jack Shirazi, Publisher Fasterj.com). Tips:

• From Java 6 you can patch an arbitrary running Java process without stopping it.
• Attach to a process using com.sun.tools.attach.VirtualMachine.attach(PID), then load an agent using com.sun.tools.attach.VirtualMachine.loadAgent().
• Access arbitary data structures by navigating from the application roots using reflection to investigate the state of those data structures.
• Redefine classes dynamically using the Instrumentation methods.

http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/TSG-VM.pdf
Troubleshooting Guide for Java SE 6 (Page last updated December 2006, Added 2007-02-27, Author Sun, Publisher Sun). Tips:

• Add -XX:+HeapDumpOnOutOfMemoryError to get a heap dump when the VM detects a native out-of-memory error.
• Use java -agentlib:hprof=<options> ... to profile an application (-agentlib:hprof=help to see the available options). HPROF is capable of presenting CPU usage, heap allocation statistics, and monitor contention profiles. In addition, it can report complete heap dumps and states of all the monitors and threads. output is normally generated when the VM exits, although this can be disabled by setting the "dump on exit" option to "n" (doe=n). In addition, a profile is generated when Ctrl-\ or Ctrl-Break (depending on platform) is pressed. On Solaris OS and Linux a profile is also generated when a QUIT signal is received (kill -QUIT pid). If Ctrl-\ or Ctrl-Break is pressed multiple times, multiple profiles are generated to the one file.
• [Article gives several example of using hprof to profile an application].
• JConsole can attach to any Java SE application in order to display useful information such as thread usage, memory consumption, and details about class loading, runtime compilation, and the operating system.
• JConsole can be used to dynamically change several parameters in the running system. For example, the setting of the -verbose:gc option can be changed so that garbage collection trace output can be dynamically enabled or disabled for a running application.
• The JConsole Overview pane displays graphs showing, heap memory usage, number of threads, number of classes, and CPU usage.
• The JConsole Memory pane displays graphs of memory usage, size, and GC information over time for selected memory areas (heap, non-heap, various memory pools).
• The JConsole Threads pane displays graphs of thread usage, live threads, peak threads, thread state and stack, and allows deadlock detection.
• The JConsole Classes pane displays graphs loaded and unloaded classes.
• The JConsole VMSummary pane displays uptime, CPU time, compile time, thread, memory, gc, finalization, operating system and class summary information, and more.
• The JConsole MBeans pane displays a tree structure showing all registered platform and application MBeans, including allowing you to invoke operations.
• The jdb tool allows you to debug a core file (as well as a java process), e.g. jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=$JAVA_HOME/bin/java,core=core.20441
• The jhat tool can browse a binary format heap snapshot e.g. a heap dump produced by jmap -dump and other techniques. jhat allows querying using OQL, standard queries reported: All classes and their instances; particular classes; individual objects; all roots; all reacable objects; new instances (if the jhat dump has more than one heap); various histograms.
• You can use jhat to determine what is keeping an object alive and where was the object allocated.
• The jhat -baseline option allows two dumps to be compared if they were produced by HPROF from the same VM instance, e.g. jhat -baseline snapshot.hprof#1 snapshot.hprof#2
• The jinfo command-line utility gets configuration information from a running Java process or a crash dump. With the -flag option, jinfo can dynamically set, unset, or change the value of some Java VM flags.
• jmap prints memory related statistics for a running VM or core file.
• jmap -heap prints heap usage, gc algorithm and heap configuration details.
• jmap -histo prints a histogram of the current heap or core (class, number of instances, and the total number of bytes used by all the instances together).
• jmap -permstat prints details of the perm space contents.
• jmap -permgen prints statistics for the objects in the permanent generation, including information about internalized String instances.
• jps prints a list of process ids for each java process running
• jstack attaches to the specified process or core file and prints the stack traces of all threads
• jstack can perform deadlock detection.
• jstack can provide a mixed stack dump of native and java stack frames.
• jstat provides information on performance and resource consumption of running applications.
• jstat -class prints statistics on the behavior of the class loader;
• jstat -compiler prints statistics of the behavior of theHotSpot compiler.
• jstat -gc prints statistics of the behavior of the garbage collected heap.
• jstat -gccapacity prints statistics of the capacities of the generations and their corresponding spaces.
• jstat -gccause prints the summary of garbage collection statistics (same as -gcutil), with the cause of the last and current (if applicable) garbage collection events.
• jstat -gcnew prints statistics of the behavior of the new generation.
• jstat -gcnewcapacity prints statistics of the sizes of the new generations and its corresponding spaces.
• jstat -gcold prints statistics of the behavior of the old and permanent generations.
• jstat -gcoldcapacity prints statistics of the sizes of the old generation.
• jstat -gcpermcapacity prints statistics of the sizes of the permanent generation.
• jstat -gcutil prints a summary of garbage collection statistics.
• jstat -printcompilation prints HotSpot compilation method statistics.
• The visualgc tool provides a graphical view of the garbage collection system.
• On Unix Ctrl-backslash and kill -QUIT, on Windows Ctrl-Break causes the HotSpot VM to print a thread and heap summary dump to the application's standard output.
• Reported thread states are: NEW - The thread has not yet started; RUNNABLE - The thread is executing in the Java virtual machine; BLOCKED - The thread is blocked waiting for a monitor lock; WAITING - The thread is waiting indefinitely for another thread to perform a particular action; TIMED_WAITING - The thread is waiting for another thread to perform an action for up to a specified waiting time; TERMINATED - The thread has exited.
• Solaris tools useful for assisting JVM analysis: coreadm, cpustat, cputrack, counters, c++filt, dtrace, gcore, intrstat, iostat, libumem, mdb, netstat, pargs, pfiles, pldd, pmap, prstat, prun, ps, psig, pstack, pstop, ptree, sar, sdtprocess, sdtperfmeter, top, trapstat, truss, vmstat, watchmalloc.
• Linux tools useful for assisting JVM analysis: c++filt, gdb, libnjamd, lsstack, ltrace, mtrace, muntrace, proc tools (pmap, pstack, etc), strace, top, vmstat.
• Windows tools useful for assisting JVM analysis: dumpchk, msdev, userdump, windbg, /Md and /Mdd compiler options.
• java.lang.Thread.getAllStackTraces returns a map of stack traces for all live threads.
• Exception in thread "main": java.lang.OutOfMemoryError: Java heap space - possibly caused by Java object retention (memory leak) or just not a large enough heap for peak usage (increase Xmx value).
• Exception in thread "main": java.lang.OutOfMemoryError: PermGen space - not enough space in the perm heap area (class objects and String.intern'ed objects). Increase the -XX:MaxPermSize value.
• Exception in thread "main": java.lang.OutOfMemoryError: Requested array size exceeds VM limit - the application tried to allocate an array that is larger than the heap size.
• Exception in thread "main": java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space? - an allocation from the native heap failed and the native heap might be close to exhaustion. Can be caused because: the OS is configured with not enough swap; or another process on the system is consuming all memory resources; or the process reached the maximum size possible for the OS (e.g. 4GB on 32-bit Unix, 2GB on most 32-bit Windows).
• Exception in thread "main": java.lang.OutOfMemoryError: <reason> <stack trace> (Native method) - a native method encountered an allocation failure, probably from running out of availble native memory. Can be caused because: the OS is configured with not enough swap; or another process on the system is consuming all memory resources; or the process reached the maximum size possible for the OS (e.g. 4GB on 32-bit Unix, 2GB on most 32-bit Windows).
• Classes with finalize methods do not have their space reclaimed at garbage collection (gc) time, instead, after gc the objects are queued for finalization, which occurs at a later time.
• Heap OutOfMemoryError can also be caused by excessive use of finalizers - if the finalizer thread cannot keep up with the finalization queue, then the Java heap could fill up and OutOfMemoryError would be thrown.
• An application that creates high-priority threads that cause the finalization queue to increase at a rate that is faster than the rate at which the finalizer thread is servicing that queue can generate an OutOfMemoryError.
• At the OS level, a malloc system call returns NULL if there is no memory available. If the return from malloc is not checked, then this can lead to memory corruption eventually leading to a crash. The cause of these crashes can be difficult to identify.
• Create a heap dump with the hprof profiler using java -agentlib:hprof=file=snapshot.hprof,format=b application and pressing Ctrl-\ or Ctrl-Break (depending on the platform) or send a QUIT signal via the kill -QUIT pid command.
• Create a heap dump with jmap using jmap -dump:format=b,file=snapshot.jmap process-pid
• Obtain a heap dump with JConsole MBeans tab using the HotSpot Diagnostic MBean dumpHeap operation.
• Specify the -XX:+HeapDumpOnOutOfMemoryError command-line option to get a heap dump when an OutOfMemoryError is thrown.
• Get a heap histogram as an initial memory profile, using jmap -histo or with a Ctrl-Break of -XX:+PrintClassHistogram is specified.
• You can get a heap histogram from a core file using jmap -histo.
• Monitor the number of objects pending finalization using the jconcole summary tab, or jmap -finalizerinfo or java.lang.management.MemoryMXBean.getObjectPendingFinalizationCount.
• Tracking native memory allocation: use the Microsoft C++ compiler /Md and /Mdd compiler options that add extra support for tracking memory allocations; Linux systems have tools such as mtrace and libnjamd to help in dealing with allocation tracking; Solaris Operating Systems provide the watchmalloc tool. Solaris has watchmalloc and libumem tools.
• Article shows examples of finding native memory leaks using dbx and separately using libumem.
• Diagnosing a looping process: get a thread dump (cntrl-break handler, or jstack, or jconsole) and focus on the threads RUNNABLE threads.
• Diagnosing deadlock: get a thread dump (cntrl-break handler, or jstack, or jconsole) and check for deadlocks.
• Diagnosing a waitleak/livelock (threads waiting on a monitor that is never notified): get a thread dump (cntrl-break handler, or jstack, or jconsole) and examine each of the threads that is blocked in Object.wait() - you will usually need application knowledge to determine the problem.
• The VM thread is normally in one of three states: waiting to execute a VM operation; synchronizing all threads in preparation for a VM operation; or executing a VMoperation. A hang that is a HotSpot VM bug rather than an application deadlock, will show in the VM thread - you may need to force a stack dump with jstack -F.
• Article includes detailed appendices on many formats including thread dumps and crash logs
• jhat can now parse incomplete and truncated heap dumps, and 64-bit heap dumps.
• com.sun.management.HotSpotDiagnostic MBean property is new.

http://www.JavaPerformanceTuning.com/news/javvathehutt075.shtml
Java 5 Upgrade Woes (Page last updated February 2007, Added 2007-02-27, Author Hutt5, Publisher JavaPerformanceTuning.com). Tips:

• Apply coding standards using automated checking to avoid bad practices creeping into your code
• Don't rely on the format produced by toString() methods, as these can change.
• -XX:+HeapDumpOnOutOfMemoryError is useful.
• Avoid serializing inner classes (even inadvertently).
• Set serialVersionUID for any class that can be serializable - and enforce that using an automated checker
• Beware of relying on classloading ordering (and doing do based on the XYZ.class idiom can be problematic if upgrading to Java 5).

Jack Shirazi Back to newsletter 075 contents