JProfiler Get rid of your performance problems and memory leaks! Modern Garbage Collection Tuning Java Performance Training Courses COURSES AVAILABLE NOW. We can provide training courses to handle all your Java performance needs Java Performance Tuning, 2nd ed The classic and most comprehensive book on tuning Java Java Performance Tuning Newsletter Your source of Java performance news. Subscribe now! Enter email: Training online Threading Essentials course JProfiler Get rid of your performance problems and memory leaks!

Back to newsletter 241 contents

This month I'm extending my special newsletter on memory leaks, adding JFR memory leak analysis and native memory leak analysis to the heap analysis focus from that previous special newsletter. The same 4 step-by-step procedure applies to all these leaks:

1. Do I have a leak (and does it needs fixing)?
   • This is the same for all types of leak, you need to track memory to see if it's increasing after Full GCs
   • On the heap side you can track heap usage via GC logs, JMX GC metrics (eg using JConsole or equivalent), JFR GC events (it's easiest to view these from JMC)
   • On the native side you can track process size with Proportional Set Size memory tracking: PSS = Private Physical Memory + Shared Physical Memory / Number of Multi-Mappers
   • If it's leaking on the native side, you'll see the heap staying roughly the same overall, but PSS increasing
   • If the leak is on the native side, you should check the various memory pools in the JMX metrics to see if it's leaking in any of these JVM managed memory areas or outside of JVM management (typically from JNI)
2. What is leaking (instances of which classes)?
   • Obviously any auto-analysis capability (such as from JMC or Eclipse MAT) makes this straightforward as it will suggest the leaking object types
   • In the absence of auto-analysis (or in addition for confirmation) a class histogram (from jmap or a heap dump or JFR) will suggest the leaking objects, but one histogram alone is usually not enough
   • Comparing two histograms separated over time (with both taken just after full GCs) should identify the objects very clearly
   • If you can reproduce the leak outside of production, any significant increases in generation counts can quickly identify leaking objects (using any profiler that shows generation counts, eg visualvm)
3. What is keeping the objects alive (what instances in the app)?
   • Again auto-analysis capability often identifies this for you, but in the absence of that you will most likely need either a heap dump or a JFR recording taken with path-to-gc-roots=true to find this
   • A native leak where the leak is from objects being kept alive in the heap that are pointing at native memory is similar to a heap leak - only more difficult to see because the heap side is quite small. But the techniques to find the leak and root holding the leak live is the same as for heap leaks
   • But with native leaks there is the additional possibility that it is an old-fashioned C-style memory leak which is memory incorrectly not reclaimed by the JNI code, in this last case no object is keeping the memory alive, it's now just memory that is no longer managed in the process space
4. Where is it leaking from (the code where the objects are created and/or assigned)?
   • To identify the code location in heap leaks that can't be reproduced outside production, you are restricted to JFR recordings, looking at "live objects" (the recording needs to be from JDK10+ as it needs Old Object Sample events)
   • For heap leaks reproducible outside production, any memory profiler with stack allocation traces (eg visualvm) can find the allocating method and even the line of code (the latter if line information hasn't been stripped from the class files)
   • For native leaks from JVM managed memory pools, you need to turn on native memory tracking eg -XX:NativeMemoryTracking=detail (can be on in production but has a 10% performance impact) and take a difference against a baseline (jcmd VM.native_memory baseline, jcmd PID VM.native_memory detail.diff)
   • For native memory leaks from JNI managed memory, you need to use a native memory profiler like jemalloc, etc.

A note from this newsletter's sponsor

News

Java performance tuning related news (not memory leak related)

• An exercise in latency reduction with some lessons learned
• Review of OpenTelemetry and how it fits into the measurement landscape using an open source implementation (the Elastic one) with a running example
• Why Java is better than C++ for high speed trading systems
• 40 milliseconds of latency from TCP_NODELAY

Tools

Java performance tuning related tools.

• JDK Mission Control Tutorial
• JVM code for examples of native leaks
• App to simulate performance problems
• New Relic open source Java agent using JFR to get JVM metrics

Articles

• Troubleshooting Native Memory Leaks in Java Applications (Poonam Parhar/Oracle)
• Troubleshooting Problems With Native (Off-Heap) Memory in Java Applications (Misha Dmitriev/Java Zone)
• Identify & Fix JVM memory leaks with Java Flight Recorder and JDK Mission Control (Cameron McKenzie/Mission Control)
• Solving Memory Leaks without Heap Dumps (Marcus Hirt/hirt.se)

Jack Shirazi

Back to newsletter 241 contents