JProfiler Get rid of your performance problems and memory leaks! Modern Garbage Collection Tuning Java Performance Training Courses COURSES AVAILABLE NOW. We can provide training courses to handle all your Java performance needs Java Performance Tuning, 2nd ed The classic and most comprehensive book on tuning Java Java Performance Tuning Newsletter Your source of Java performance news. Subscribe now! Enter email: Training online Threading Essentials course JProfiler Get rid of your performance problems and memory leaks!

Back to newsletter 269 contents

https://serce.me/posts/01-02-2023-jvm-field-guide-memory
JVM Field Guide: Memory (Page last updated February 2023, Added 2023-04-30, Author Sergey Tselovalnikov, Publisher serce.me). Tips:

• Virtual memory is the amount of native memory that the JVM could use, typically set by -Xmx; resident memory is the amount of native memory that the JVM is actually using.
• If JVM memory usage, including both the heap and off-heap memory used, exceeds the container memory limits, the JVM will be killed either with an OutOfMemory error, or by the OOM killer. There is also the common situation that the JVM doesn't die but just cycles through multiple GCs while the application makes almost no progress, because heap limits have been reached but not yet exceeded.
• -XX:+AlwaysPreTouch ensures that requested memory is allocated at JVM initialization. This is only useful for systems where you need to minimize native memory allocation and paging overheads, while at the same time there is no competition at the OS level for those memory resources.
• Finding the cause of memory leaks can be found by a heap dump, though a heap histogram (eg via jmap -histo ...) may give hints and is lower cost.
• Making a GC choice is always a tradeoff between pause times and the CPU/memory overhead of the collector. An example of GC logging is -Xlog:safepoint*,gc*:file=gc/gc.log:time,level,tags:filecount=5,filesize=5000k.
• Safepoint data can often be overlooked when analysing application pauses, but their impact can be significant. Useful options include -Xlog:safepoint* -XX:+SafepointTimeout -XX:SafepointTimeoutDelay.
• Sometimes the only way to achieve the desired performance is to reduce the number of allocations. For this, allocation profiling is needed to identify the source of high allocations.
• Native memory tracking can be enabled with -XX:NativeMemoryTracking=summary, and a common cause of native memory leaks is class generation. In this case limiting metaspace -XX:MaxMetaspaceSize can prevent the host from running out of resources (though the JVM will still hit an OutOfMemroy error).

https://www.youtube.com/watch?v=UhfUNqFDzog
Java is Very Fast, If You Don't Create Too Many Objects (Page last updated October 2022, Added 2023-04-30, Author Peter Lawrey, Publisher Engineers.SG). Tips:

• Increasing levels of abstraction adds overhead exponentially.
• Allocations don't scale well. In single threaded tests, allocation is very quick. But most production systems are multi-threaded and multi-threaded allocations at a high rate (as you have in bursts of activity) has concurrency bottlenecks such that the latency from allocations can be many times the latency impact of GCs!
• Allocation rates of around 10GB/s is the maximum that the majority of machines can sustain - regardless of core counts. And at that rate it dominates CPU. Four threads allocating is already enough to hit the highest per-thread allocation rate possible on many systems. The L3 cache is the limiting factor.
• Base 85 will store 10 of the 85 most common characters into a long - that's very efficient because it's not an additional object, and comparison is very efficient. Date/Times stored as longs are also very efficient.
• Storing data (datastores and persisted/reliable messaging) in the critical path is usually the biggest bottleneck.
• Zero garbage is not a useful target, you are better off targeting sufficiently minimum garbage that the heap fills slowly enough to miss GCs at critical times.

https://www.youtube.com/watch?v=NI16bqeGv7U
Troubleshooting Native Memory Issues in Java Applications (Page last updated March 2023, Added 2023-04-30, Author Poonam Parhar, Publisher Java). Tips:

• A native out-of-memory error could be from other processes using up the system memory. You can remove or reduce their memory usage, and/or reduce the heap used on the JVM to make more native memory available.
• JVM memory includes: the heap, metaspace (classes and metadata), codecache (both bytecode and compiled code), JVM management space, loaded jars and native libraries, thread stacks and thread-local storage, JNI/JVMTI allocated memory, NIO allocations, direct bytebuffers.
• To confirm there is a native memory leak, you need to monitor the resident set size of the JVM (or proportional set size if using multi-mapping). Look for consistently increasing memory size, then eliminate that it is a heap memory leak by comparing that against the process memory size as it changes.
• Native memory tracking (-XX:NativeMemoryTracking=summary / -XX:NativeMemoryTracking=detail) can be used to look at native memory that the JVM knows about. It causes a 5%-10% performance impact. The reports can be obtained with jcmd using VM.native_memory [baseline, detail.diff, summary.diff].
• For native memory tracking of memory the JVM doesn't manage, you need to use native memory tracking tools like jemalloc, valgrind, dbx, purify, pmap, and core dump files.
• The overall process for native memory leak analysis is: 1. determine that the machine/container has enough native memory available for the expected memory usage; 2. Check that the JVM process memory is consistently increasing; 3. force garbage collections and eliminate heap memory as the cause (or fix that); 4. eliminate JVM managed native memory as the cause using NativeMemoryTracking (or fix that); 5. use a native memory profiler or dump analysis to find the cause of native memory leaking outside of the JVM managed native memory.
• Common native memory leak scenarios: using compressed oops conflicting with the space available in low memory base (fix with -XX:HeapBaseMinAdress=n); direct byte buffers (make sure the Java wrapper is released); NIO APIs (make sure the Java wrapper is released); inflaters/deflaters (make sure you end() them); finalizers (make sure the finalizer processing thread has run).

Jack Shirazi Back to newsletter 269 contents