JProfiler Get rid of your performance problems and memory leaks! Modern Garbage Collection Tuning Java Performance Training Courses COURSES AVAILABLE NOW. We can provide training courses to handle all your Java performance needs Java Performance Tuning, 2nd ed The classic and most comprehensive book on tuning Java Java Performance Tuning Newsletter Your source of Java performance news. Subscribe now! Enter email: Training online Threading Essentials course JProfiler Get rid of your performance problems and memory leaks!

Back to newsletter 208 contents

https://www.youtube.com/watch?v=t0HkM7g5bxA
Java in a World of Containers (Page last updated September 2017, Added 2018-03-27, Author Paul Sandoz, Publisher Oracle). Tips:

• JDK 9 respects cgroup memory limits: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap .
• JDK 9 respects CPU constraints: Runtime.availableProcessors now reports the CPUs available to the container rather than the host, and the GC (at least G1) respects this too
• JDK 9 jlink tool let's you create custom Java runtimes, eg just java.base [the talk shows an example of building a custom docker image with a custom runtime containing only java.base]
• The Portola project ports the JDK for Alpine Linux

https://jelastic.com/blog/java-ram-usage-in-containers-top-5-tips-not-to-lose-your-memory/
Java RAM Usage in Containers: Top 5 Tips Not to Lose Your Memory (Page last updated April 2017, Added 2018-03-27, Author Tetiana Fydorenchyk, Publisher Jelastic). Tips:

• If the Xmx option is not defined explicitly, the JVM tries to use 1/4th of all memory available for the host OS which can lead to the container killing the Java process if the JVM memory usage grows over the cgroups limit defined for a Docker container. To avoid this, -XX:+UseCGroupMemoryLimitForHeap automatically sets Xmx to the memory limit defined in cgroup.
• For JDK8+, class names and fields, method bytecode, the constant pool, etc are now located in Metaspace which is stored outside of JVM heap. GC and JIT also use non-heap native memory, as does DirectByteBuffer and JNI. Limit the Metaspace size with -XX:MaxMetaspaceSize. You can track native memory allocation with -XX:NativeMemoryTracking=summary but this will incur a 5-10% performance hit.
• MinHeapFreeRatio and MaxHeapFreeRatio are manageable (from JDK7u60+), so you can change their values in runtime without the need to restart the Java process. This is useful for minimzing JVM memory in flight.
• Use -XX:-ShrinkHeapInSteps to disable some GC cycles and release unused RAM resources faster and minimize the Java heap size usage in applications with variable load (will change the GC profile though, giving some larger pauses).
• Trigger a full GC before checkpointing the image to disk as that will minimize the saved image footprint.

https://www.youtube.com/watch?v=s3NZcKtg-5c
Java in a World of Containers (Page last updated February 2018, Added 2018-03-27, Author Mikael Vidstedt, Matthew Gilliard, Publisher FOSDEM). Tips:

• A docker image with the full JDK is around 0.5GB. With JDK9+ you can create custom JREs with just the subset of the JDK you need (jdeps tool helps identify which modules an application uses). A minimal JRE containing only the java.base module, eg ' jlink --module-path /docker-java-home/jmods --strip-debug --compress=2 --output java --add-modules java.base' with alpine linux and the musl-libc library in the image results in an image of less than 50MB in size (an order of magnitude smaller). (This is maintained in project Portola)
• Class data sharing - UseAppCDS (see http://www.javaperformancetuning.com/news/news207.shtml for details) - works nicely for multiple instances of the same container, if you put the archive in the shared image.
• The (experimental) AOT compiler jaotc can help reduce footprint and speed startup, but probably too early still in Java 10 to use.
• Support for honouring cgroups limits set by the container has been added to Java 10 and cpu limitations are now supported by Runtime.availableProcessors(), ForkJoin pool, VM internal thread pools, core.async, ElasticSearch, Netty.
• Additional flags that let you choose the % of RAM available to the container -XX:InitialRAMPercentage -XX:MaxRAMPercentage -XX:MinRAMPercentage

https://www.youtube.com/watch?v=bK9A5ODIgac
Container Performance Analysis (Page last updated April 2017, Added 2018-03-27, Author Brendan Gregg, Publisher Docker). Tips:

• Use an orchestration tool (like kubernetes) to autodeploy and autoscale.
• Process IDs in the container map to ones in the host and you need to know that if you are using host level monitoring and container level logging.
• The container's CPU limit = container's shares / total busy shares, eg if container X has 100 shares and there are 200 shares busy, container X gets 50% of the CPU (lets the container burst)
• The container's minimum CPU limit = container's shares / total allocated shares. This is worst case. If container X is above minimum and another tenant starts to take more CPU, container X can suddenly have less available CPU. This makes analysis tricky, since you tend to notice the decrease in performance of container X but the machine appears stably utilized.
• Overlay networks can come with a significant performance cost
• For performance analysis, you have one kernel, but 2 perspectives (container vs host) and namespaces and cgroups.
• Apply the USE method for performance analysis. For every resource check: Utilization (eg CPU time busy, cgroups % of cap); Saturation (eg CPU run queue or request latency, cgroups number of times throttled); Errors (eg CPU ECC errors). You can use this for both hardware and software resources. USE methodology limits the huge number of metrics that you can get, letting you ask what you should measure.
• Host performance analysis in 60 seconds: uptime (load averages); dmesg | tail (kernel errors); vmstat 1 (overall stats by time); mpstat -P ALL 1 (CPU balance); pidstat 1 (process usage); iostat -xz 1 (disk IO); free -m (memory usage); sar -n DEV 1 (network IO); sar -n TCP,ETCP 1 (TCP stats); top (overview)
• USE CPU: Utilization ('mpstat - P ALL 1', sum non-idle fields); Saturation ('vmstat 1', "r"); Errors (perf)
• USE Memory Capacity: Utilization ('free -m' "used"/"total"); Saturation ('vmstat 1', "si"+"so"; 'demsg | grep killed); Errors ('demsg')
• USE Storage IO: Utilization ('iostat -xz 1', "%util"); Saturation ('iostat -xnz 1', "avgqu-sz">1); Errors (/sys/.../ioerr_cnt; 'smartctl')
• USE Network: Utilization ('nicstat', "%Util"); Saturation ('ifconfig', "overrunns"; 'netstat -s' "retrans..."); Errors ('ifconfig', "errors")
• zfsslower tool identifies ZFS operations slower than a threshold, eg 1 millisecond
• iosnoop tool gets disk IO events with latency (obtain from perf-tools or biosnoop in bcc/BPF)
• btrfsdist provides latency histograms for disk IO
• 'systemd-cgtop' is a "top" for scgroups, and 'docker stats' is a "top" for docker containers; also 'nsenter -t PID -m -p top'
• Hot system process to docker container mapping: find the process ID (PID) then 'grep PID /sys/fs/cgroup/cpu,cpuacct/docker/*/tasks | cut -d/ -f7' ; also 'cat /proc/PID/cgroup'
• Find namespace with 'ls -l /proc/PID/ns/*'
• Diagnosing container performance problems: start from the possible answers and use the metrics to identify between them. Eg not enough CPU could be a) physical CPU throttled b) container cap throttled c) shares throttled d) not throttled. CPU bottleneck identification flowchart: is throttled_time increasing - true=>cap throttled, false=> nonvoluntary context switches increasing - false=>not throttled, true=>host has idle CPU - true=>not throttled, false=>all other tenants idle true=>physical CPU throttled, false=>share throttled

Jack Shirazi

Back to newsletter 208 contents