Back to newsletter 105 contents

http://codecrawler.blogspot.com/2009/05/java-multithreaded-web-service.html
Java Multithreaded Web Service Performance Tips (Page last updated May 2009, Added 2009-08-31, Author Rommel Garcia, Publisher codecrawler). Tips:

• The ScheduledExecutorService implements ExecutorService and is of type Executor. Use Executor for abstracting threads implemention into simpler methods.
• ScheduledExecutorService allows you to provide what the initial time is to run the thread associated with it, at what frequency, the thread to run, the unit of time and you can configure the thread pool size as well.
• ConcurrentLinkedQueue doesn't need the synchronized keyword to use by default it is already thread-safe.
• Using the Runnable interface is better than just extending the Thread class as it promotes less coupling.
• You can force shutdown on all threads in a ThreadPool by calling the method shutdownNow().
• A "request" scope in your wsdd file creates one instance of the service per request, spawning a thread per request. If the scope is "application" it will create only one instance of the service that will serve multiple requests.

https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4737
A simple way to analyze thread contention problems in Java (Page last updated October 2006, Added 2009-08-31, Author Markus Kohler, Publisher SAP). Tips:

• Take regular thread dumps and analyse them for contention.
• Use the command sequence 'grep "waiting to lock" dumpfile | sort -k2 -r | uniq -c |sort -k1 -n -r | more' or similar to find the the methods that are waiting for a lock
• Use the command sequence 'grep " locked" dumpfile | sort -k3 | uniq -c | sort -k 1 -n -r | more' or similar to find the the methods that are locked the longest
• If a method is frequently waiting for a lock and also found locked in the dumps, it is likely to be a thread contention bottleneck.

http://www.ibm.com/developerworks/aix/library/j-nativememory-aix/index.html
Understanding how the JVM uses native memory on AIX (Page last updated April 2009, Added 2009-08-31, Author Andrew Hall, Publisher IBM). Tips:

• When running on a system with more process address space than physical memory, a memory leak or excessive use of native memory will force the OS to swap out some of the virtual address space. Accessing a memory address that has been swapped is a lot slower than reading a resident (physical memory) address because it must be loaded from the hard drive.
• If you try to use so much RAM-backed virtual memory that your data cannot be held in physical memory, the system will thrash (spend most of its time copying memory back and forth from swap space) and performance becomes so poor the user can't fail to notice there's a problem.
• When a JVM's Java heap is swapped out, the garbage collector's performance becomes so poor that the application may appear to hang.
• If multiple Java runtimes are in use on a single machine at the same time, the physical memory must be sufficient to fit all of the Java heaps.
• Although the logical heap (the area of memory that is actively used) will grow and shrink according to the number of objects on the heap and the amount of time spent in garbage collection, the amount of native memory used remains constant and is dictated by the -Xmx value: the maximum heap size.
• The JVM memory manager (Sun & AIX) relies on the heap being a contiguous slab of memory, so it's impossible to allocate more native memory when the heap needs to expand; all heap memory must be reserved up front.
• Reserving native memory is not the same as allocating it. When native memory is reserved, it is not backed with physical memory or other storage. Although reserving chunks of the address space will not exhaust physical resources, it does prevent that memory from being used for other purposes. A leak caused by reserving memory that is never used is just as serious as leaking allocated memory.
• It's quite likely that using shared classes will occupy more (virtual) address space but less physical memory than running without shared classes.
• Each classloader has a native-memory overhead - so many classloaders each loading one class uses more native memory than one classloader that loads many classes.
• It's impossible to unload a single class; classloaders are unloaded instead, taking all the classes they loaded with them.
• A classloader can only be unloaded only if there are no references to that ClassLoader object nor to any Class objects loaded by that classloader and no objects of any class loaded by that classloader are alive.
• The three default classloaders that the Java runtime creates for all Java applications - bootstrap, extension, and application - can never be released (so neither can any class they load).
• JSP generates a class for each .jsp page executed that will last the lifetime of the classloader that loaded them - typically the lifetime of the Web application. Java reflection also generates classes.
• You can control the reflection accessor behaviour using system properties. The default inflation threshold (the number of times a JNI accessor is used before being inflated into a bytecode accessor) for Java 5.0 is 15. You can modify this by setting the sun.reflect.inflationThreshold system property. If you set the inflationThreshold to 0 or less, then the accessors will never be inflated. This can be useful if you find that your application is creating many sun.reflect.DelegatingClassloaders.
• Setting -Dsun.reflect.noInflation=true disables inflation entirely but, counterintuitively, causes bytecode accessors to be used for everything. Using -Dsun.reflect.noInflation=true increases the amount of address space consumed by reflection classloaders because many more are created.
• The GetTypeArrayElements and GetTypeArrayRegion functions can copy Java heap data into native memory buffers for the native code to work with.
• NIO added support for directByteBuffers (allocated using the java.nio.ByteBuffer.allocateDirect() method) that are backed by native memory rather than Java heap. Direct ByteBuffers can be passed directly to native OS library functions for performing I/O, making them significantly faster in some scenarios because they can avoid copying between Java heap and native heap.
• Using many or large native buffers can fill native memory without filling the java heap, causing the system memory to become exhausted by Java, even though there is space in the Java heap.
• Each java thread uses memory (native and Java). Although the amount of memory used per thread is relatively small, an application with several hundred threads could use quite a lot. Running an application with many more threads than available processors to run them is usually inefficient and can result in poor performance as well as increased memory usage.
• There is no one simple way to identify native-memory exhaustion. [The article shows a number of examples of running out of native memory].
• [Article provides examples of diagnosing naive memory problems for IBM Java on AIX.]

http://www.ibm.com/developerworks/linux/library/j-nativememory-linux/index.html
Understanding how the JVM uses native memory on Windows and Linux (Page last updated April 2009, Added 2009-08-31, Author Andrew Hall, Publisher IBM). Tips:

• When running on a system with more process address space than physical memory, a memory leak or excessive use of native memory will force the OS to swap out some of the virtual address space. Accessing a memory address that has been swapped is a lot slower than reading a resident (physical memory) address because it must be loaded from the hard drive.
• If you try to use so much RAM-backed virtual memory that your data cannot be held in physical memory, the system will thrash (spend most of its time copying memory back and forth from swap space) and performance becomes so poor the user can't fail to notice there's a problem.
• When a JVM's Java heap is swapped out, the garbage collector's performance becomes so poor that the application may appear to hang.
• If multiple Java runtimes are in use on a single machine at the same time, the physical memory must be sufficient to fit all of the Java heaps.
• Although the logical heap (the area of memory that is actively used) will grow and shrink according to the number of objects on the heap and the amount of time spent in garbage collection, the amount of native memory used remains constant and is dictated by the -Xmx value: the maximum heap size.
• The JVM memory manager (Sun & AIX) relies on the heap being a contiguous slab of memory, so it's impossible to allocate more native memory when the heap needs to expand; all heap memory must be reserved up front.
• Reserving native memory is not the same as allocating it. When native memory is reserved, it is not backed with physical memory or other storage. Although reserving chunks of the address space will not exhaust physical resources, it does prevent that memory from being used for other purposes. A leak caused by reserving memory that is never used is just as serious as leaking allocated memory.
• It's quite likely that using shared classes will occupy more (virtual) address space but less physical memory than running without shared classes.
• Each classloader has a native-memory overhead - so many classloaders each loading one class uses more native memory than one classloader that loads many classes.
• It's impossible to unload a single class; classloaders are unloaded instead, taking all the classes they loaded with them.
• A classloader can only be unloaded only if there are no references to that ClassLoader object nor to any Class objects loaded by that classloader and no objects of any class loaded by that classloader are alive.
• The three default classloaders that the Java runtime creates for all Java applications - bootstrap, extension, and application - can never be released (so neither can any class they load).
• JSP generates a class for each .jsp page executed that will last the lifetime of the classloader that loaded them - typically the lifetime of the Web application. Java reflection also generates classes.
• You can control the reflection accessor behaviour using system properties. The default inflation threshold (the number of times a JNI accessor is used before being inflated into a bytecode accessor) for Java 5.0 is 15. You can modify this by setting the sun.reflect.inflationThreshold system property. If you set the inflationThreshold to 0 or less, then the accessors will never be inflated. This can be useful if you find that your application is creating many sun.reflect.DelegatingClassloaders.
• Setting -Dsun.reflect.noInflation=true disables inflation entirely but, counterintuitively, causes bytecode accessors to be used for everything. Using -Dsun.reflect.noInflation=true increases the amount of address space consumed by reflection classloaders because many more are created.
• The GetTypeArrayElements and GetTypeArrayRegion functions can copy Java heap data into native memory buffers for the native code to work with.
• NIO added support for directByteBuffers (allocated using the java.nio.ByteBuffer.allocateDirect() method) that are backed by native memory rather than Java heap. Direct ByteBuffers can be passed directly to native OS library functions for performing I/O, making them significantly faster in some scenarios because they can avoid copying between Java heap and native heap.
• Using many or large native buffers can fill native memory without filling the java heap, causing the system memory to become exhausted by Java, even though there is space in the Java heap.
• Each java thread uses memory (native and Java). Although the amount of memory used per thread is relatively small, an application with several hundred threads could use quite a lot. Running an application with many more threads than available processors to run them is usually inefficient and can result in poor performance as well as increased memory usage.
• There is no one simple way to identify native-memory exhaustion. [The article shows a number of examples of running out of native memory].
• [Article provides examples of diagnosing naive memory problems for Java on Windows and Linux.]

http://www.ibm.com/developerworks/aix/library/au-aixperformancetuning/index.html
Advanced performance tuning concepts (Page last updated April 2009, Added 2009-08-31, Author Sean Walberg, Publisher IBM). Tips:

• The primary aspects of the system that affect performance are: CPU; Memory; Disk space; Disk speed; Network.
• On Unix vmstat gives a good real-time breakdown of where a system is spending its time, and the sar suite of tools is good for longer term monitoring.
• If monitoring shows that your CPU is spending most of its time in user space and the idle cycles are dwindling, it is time to either make the application more efficient or shuffle load to another server or increase the CPU capacity.
• Ensure that you have enough RAM to hold your application without needing to use swap space.
• Most Unix systems try to use free memory for caching, which is why it often looks like your system has no free memory - take this into account when measuring free memory and virtual memory usage.
• Excessive disk activity is usually a bad sign. Disk activity can be due to swapping or it can be due to requests from the application or operating system. Excessive logging can cause contention on a disk.
• On Unix Use iostat to tell you how many reads and writes are happening at a particular point in time and how saturated your disk controllers are.
• If you have several disks, splitting load onto separate spindles is an effective way of making reads and writes faster because the largest component of disk latency is the seek time.
• Files that grow consistently, such as log files and database journals, should be on separate disks from your application's disk and database.
• A high iowait value (CPU is idle but the system is waiting for IO to return) can be an indication of a slow or overloaded disk - both vmstat and iostat report this statistic.
• If you run out of file descriptors, attempts to open files will fail. Usually, the ulimit command is enough to increase the number of available file descriptors, though your operating system may have kernel limitations preventing the ulimit from succeeding.
• Make sure all servers are set to the highest network speed possible using full duplex and that the switch port is set to match.
• Look for signs of buffer congestion with netstat -s, which prints a list of network counters. Anything with the words "queue" or "overflow" in it is related to the TCP queues and should be monitored. These counters are generally reset only at boot, so you are more concerned with numbers that grow over time.
• A significant amount of Java tuning involves determining the optimal memory sizes for the heap and fine tuning the garbage collection process.
• The rough idea behind tuning the garbage collection process is to understand how often it is run and the conditions that cause it to run, and then change the JVM settings to minimize the impact of the garbage collection run.
• Enable verbose garbage collection logging in your JVM with the -verbose:gc parameter.
• Use the steady state value of the heap size as your initial size setting.
• Use verbose garbage collection logging to determine the percentage of time the JVM spends in garbage collection, then tune to minimize this.
• Tuning recommendations in priority order are: make sure your server has the CPU, disk, memory, and network resources it needs; tune the garbage collection to minimize time spent in garbage collection; make sure your request handling queues are tuned to handle the requests throughput the system can manage - excess request should be queued prior to entry in the Java request processing system so that the system doesn't get overloaded; size caches so that they are not contantly being purged to make room for new entries.

http://jeremymanson.blogspot.com/2009/07/how-hotspot-decides-to-clear_07.html
How Hotspot Decides to Clear SoftReferences (Page last updated July 2009, Added 2009-08-31, Author Jeremy Manson, Publisher jeremymanson). Tips:

• A SoftReference is a reference that the garbage collector can decide to clear if it is the only reference left to an object, and the GC decides to clear it.
• A WeakReference is a reference that the garbage collector will aggressively clear if it is the only reference left to the object.
• SoftReferences are generally used to implement memory-sensitive caches.
• In the Sun HotSpot JVM, softreferences are cleared according to how long since they were last accessed and how much free space there is in the heap.
• Adjusting -XX:SoftRefLRUPolicyMSPerMB=n down (from the default 1000) makes SoftReferences more likely to be cleared quicker; adjusted up and you get the opposite effect.

Jack Shirazi Back to newsletter 105 contents