JProfiler Get rid of your performance problems and memory leaks! Modern Garbage Collection Tuning Java Performance Training Courses COURSES AVAILABLE NOW. We can provide training courses to handle all your Java performance needs Java Performance Tuning, 2nd ed The classic and most comprehensive book on tuning Java Java Performance Tuning Newsletter Your source of Java performance news. Subscribe now! Enter email: Training online Threading Essentials course JProfiler Get rid of your performance problems and memory leaks!

Back to newsletter 241 contents

https://blogs.oracle.com/poonam/troubleshooting-native-memory-leaks-in-java-applications
Troubleshooting Native Memory Leaks in Java Applications (Page last updated December 2020, Added 2020-12-28, Author Poonam Parhar, Publisher Oracle). Tips:

• The OutOfMemoryErrors related to "Out of swap space?" and "unable to create new native Thread" indicate that the JVM has run out of native memory resources as opposed to JVM heap resources
• Ways to free up more native space for a JVM include: reducing the size of the JVM heap; reducing the size of Metaspace; reducing the number of threads; reducing the size available to thread stacks; and even terminating other processes on the system that are not needed.
• JVM memory spaces include: The object heap; Metaspace (classes, metadata); the Codecache (JIT compilation and compiled code); memory for Internal JVM operations; memory used by native libraries and jars; thread stacks; thread-local storage; NIO allocations; direct ByteBuffers; JNI and JVMTI memory allocations.
• Monitor native memory by tracking Proportional Set Size (PSS = Private Physical Memory + Shared Physical Memory / Number of Multi-Mappers; on Linux this is available from the "Pss:" line in /proc/$pid/smaps). If it's leaking on the native side, you'll see heap not increasing but PSS increasing (make sure to force full GCs before each measurement, so that any garbage objects holding on to memory - both heap and native - has first been cleared).
• For native memory leaks from JVM managed memory spaces, use Native Memory Tracking (can result in a 5 to 10 percent JVM performance drop): -XX:NativeMemoryTracking=summary or -XX:NativeMemoryTracking=detail ; jcmd VM.native_memory baseline and jcmd VM.native_memory detail.diff/summary.diff gives you access to the information.
• For native memory leaks from memory outside JVM managed memory, typically caused by JNI or native libraries, you need to use a 3rd party tool to find the leak, eg jemalloc, valgrind, dbx, purify, UMDH, pmap and core files [the article includes examples of using these tools to identify a native memory leak].
• There is an edge case where compressed oops can cause an apparent native memory exhaustion, in this case use -XX:-UseCompressedOops [details at https://blogs.oracle.com/poonam/running-on-a-64bit-platform-and-still-running-out-of-memory ]

https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo
Troubleshooting Problems With Native (Off-Heap) Memory in Java Applications (Page last updated July 2018, Added 2020-12-28, Author Misha Dmitriev, Publisher Java Zone). Tips:

• ByteBuffer.allocate() (which creates a HeapByteBuffer) has memory allocated in the heap, while ByteBuffer.allocateDirect() (which creates a DirectByteBuffer) has memory allocated in native memory. If you use a HeapByteBuffer, OS I/O calls have to copy the data to and from the buffer; using a DirectByteBuffer allows the OS I/O calls to operate directly on the native buffer so copying can be avoided.
• DirectByteBuffer (created by ByteBuffer.allocateDirect()) memory is allocated in native space, not the object heap, and released when the finalize method is called (usually by the garbage collector when the DirectByteBuffer pointing at the native memory is reclaimed).
• "OutOfMemoryError: Direct buffer memory" is indicative of either insufficient memory allocated for your direct buffers, in which case either increase the -Xmx value (the direct buffer memory is not allocated in the heap, but by default has the same additional size available) or the -XX:MaxDirectMemorySize value if you want to keep -Xmx sizing separate; or if you think there should be sufficient direct buffer memory from the already allocated size and suspect a leak, set the -Djdk.nio.maxCachedBufferSize which covers the most common direct buffer leak of direct buffers being cached in thread-locals. Note for the latter, the size is per-thread so you need to factor that in to your sizing, it will depend on how many threads you have using direct buffers.

https://www.youtube.com/watch?v=DM8hiMrQB7g
Identify & Fix JVM memory leaks with Java Flight Recorder and JDK Mission Control (Page last updated December 2020, Added 2020-12-28, Author Cameron McKenzie, Publisher Mission Control). Tips:

• Use jdk flightrecorder to record the application runtime and jdk mission control to view the recordings.
• Using jdk mission control to analyse a memory leak: the automated analysis results can provide auto-suggested leak candidates for memory leaks; check the memory tab to see if the memory usage is increasing; check the same tab to see the histogram of objects; the "live objects" sub-tab of the memory tab can (possibly) identify the objects that are leaking (by using the OldObjectSampleEvent)

http://hirt.se/blog/?p=1055
Solving Memory Leaks without Heap Dumps (Page last updated November 2018, Added 2020-12-28, Author Marcus Hirt, Publisher hirt.se). Tips:

• There are many situations where you can't or don't want to take a heap dump - or where even if you could and wanted to, the heap would be too big to analyze. An alternative is to use Java Flight Recorder to find the leak.
• The Old Object Sample event was introduced in JDK 10 and tracks a fixed number of objects on the heap, for as long as they are live. The objects are selected upon retiring a TLAB, or when allocating outside of TLABs. The details stored include allocation time, allocation stack trace, thread id, type of object, memory address and array size (if it's an array). The samples are stored in a fixed size queue, with weak references to the samples - if a sampled object is garbage collected, it is removed. The paths back to the GC roots can also be calculated if enabled for the recording using -XX:StartFlightRecording=path-to-gc-roots=true or jcmd PID JFR.dump path-to-gc-roots=true
• The JMC algorithm for selecting good leak candidates is: first check if there is an increasing live set; if so use the Old Object Samples to find good candidates using a combination of the distance from the root, the ratio of how many objects this candidate keeps alive to how many objects its root keeps alive and the ratio of how many objects the candidate keeps alive to how many objects are alive globally (the implementation is in ReferenceTreeModel in the JMC project).

Jack Shirazi Back to newsletter 241 contents