Back to newsletter 096 contents

http://www.kodewerk.com/consumers_of_the_cpu.htm
Consumers of the CPU (Page last updated October 2008, Added 2008-11-27, Author Kirk Pepperdine, Publisher kodewerk). Tips:

• Breakdown CPU usage break into operating system, JVM, and application level usage. Typically you want the application using the CPU rather than OS or JVM.
• To monitor OS level CPU usage, vmstat or taskmgr is pretty much all you need. With taskmgr be sure that you have the boxed ticked that say show kernel times (it will show up as a red line in the CPU utilization graph).
• OS level CPU usage should be a small percentage of overall CPU utilization, less than 5% on average, for typical applications. (Some applications, especially I/O intensive ones, will have a much higher OS CPU utilization).
• JVM ex-application CPU utilization is difficult to determine. The most common task that a JVM performs is memory management, so a good proxy for JVM-level CPU utilization is garbage collection (a CPU intensive task) using -verbose:gc to determine GC efficiency. One should also look at GC frequency and long pause times and object churn.
• If object churn rates are too high and your application is releasing objects very quickly, than you may be in a situation where your application is bottlenecked on object creation. This will not show up in the GC efficiency calculation.
• If the OS and JVM heap management are not taking significant amounts of CPU, then only the application is left. To make it faster, you need to either improve hardware or tune.
• Generally adding more hardware to speed up your application is a cheaper option than tuning it. However we have also experienced a number of situations where more hardware made things worse.
• If the OS is determined to be a bottleneck then you have to ask what you are doing to cause the OS to work so hard. This will involve your application doing something that will result in a lot of extra calls into the OS. Think of things like looping over I/O (disk or sockets) calls or hard use of locks causing the OS scheduler to work harder than it normally would.
• A thread profiler should be employed to answer the question: what are the threads asking the OS to do?
• If heap management is a problem, then start with configuring the heap to cope with the load. If that doesn't work or if object churn is a major component of the problem, you are going to have to use a memory/object creation profiler to determine which parts of the code are responsible. Choosing an alternative algothm is the only path to relief in this case.
• If the application is the primary consumer of CPU then you are looking at straight execution profiling. Use the profiler to identify hot spots in the code and then determine steps that can be taken to reduce the strength of the algorthm.

http://java.sys-con.com/node/633761
Diagnosing Performance Issues in Production Java Applications (Page last updated August 2008, Added 2008-11-27, Author Rajagopal Marripalli, Publisher JDJ). Tips:

• Java Management Extensions (JMX)-based tools have low overhead and are often used to expose vital JVM statistics and application-specific metrics. They can provide limited visibility based on the kind of metrics exposed by the application and underlying Java container. But they cannot detail root causes of problems they expose.
• Bytecode instrumentation (BCI) tools work with the compiled Java code to insert monitoring sensors into running applications. They can provide root cause indicators, but production environments impose a limit on the number of instrument points because instrumenting each method causes additional performance overhead.
• JVMTI based tools can provide detailed information on the causes of performance issues. However even the most optimized profiling tools based on JVMTI have an overhead of nearly 10%.
• An approach of a native agent embedded in the JVM potentially allows for the lowest overhead monitoring with the most detailed information extractable.

http://www.javaspecialists.eu/archive/Issue162.html
Exceptions in Java (Page last updated June 2008, Added 2008-11-27, Author Dr. Heinz M. Kabutz, Publisher The Java Specialists' Newsletter). Tips:

• An asynchronous exception can happen anywhere in your program, even inside synchronized blocks. Asynchronous exceptions have been deprecated for a good reason, so it is advised not to use them.
• Exceptions should be thrown as early as possible.
• To avoid a ConcurrentModificationException caused by another thread modifying the collection concurrently, you need to consider all the places in your code that are modifying the collection. It is usually easier to change the collection to be threadsafe, such as CopyOnWriteArrayList, rather than to try to fix the actual problem. Another alternative is to synchronize around the entire iteration, but that comes at a great cost to concurrency.
• One solution to fix concurrent read and writes without reducing concurrency on the read is to wrap the read in aReadWriteLock.readLock().lock(); try {READ} finally {aReadWriteLock.readLock().unlock();} and the associated writes similarly aReadWriteLock.writeLock().lock(); try {WRITE} finally {aReadWriteLock.writeLock().unlock();}.
• Don't use exceptions for flow control.

http://www.ddj.com/java/210604244
Weak References as Object Accessors (Page last updated September 2008, Added 2008-11-27, Author Sergey Babkin, Publisher DrDobbs). Tips:

• finalize() methods may get delayed for an unknown length of time, until the garbage collector decides to do its sweep - for this reason expensive resources like file descriptors should be freed explicitly.
• A weak reference refers to an object but doesn't prevent this object from being garbage collected if all the normal strong references to it are gone and the system starts running short of memory (after a garbage collection the weak reference would contain null).

http://www.infoq.com/articles/Wait-Based-Tuning-Steven-Haines
A Formal Performance Tuning Methodology: Wait-Based Tuning (Page last updated October 2008, Added 2008-11-27, Author Steven Haines, Publisher InfoQ). Tips:

• Performance tuning can be summarized simply in four steps: Load Test; Container Tuning; Application Tuning; Iterate.
• To start performance tuning you need proper load test suite. A load test must be representative of what users are doing.
• An unbalanced load test can result in tuning the wrong portions of an application, wasting effort and missing the real bottlenecks. However user behavior is inferred, it is a core prerequisite before starting any performance tuning exercise.
• Validate the load test suite by comparing load test application activity with production activity.
• Tuning the cache hit ratio to 80% might be a good thing, but first ask: How dependent is the application on the cache; How important are these cache requests compared to others in the application; Should the items be cached at all?
• Performance tuning should be performed in the context of the architecture of the application being tuned, rather than just targeted at arbitrary bottlenecks.
• Identify the core processing paths in an application: include all tiers that a request may pass between, all external services that the request may interact with, all objects that are pooled, and all objects that are cached.
• The size of each thread pool must be tuned considerating: sized large enough so that incoming requests do not need to wait unnecessarily; not so large that it saturates the server (context switching is unnecessary overhead); sized to not saturate any backend resources.
• The optimal size for a server thread pool is the number of threads that generate maximum load without overloading any limiting resources.
• Don't just tune JDBC connection pools to send the most optimal amount of load to the database, you should review the SQL being executed and tune that.
• Common technology-based wait points are: Pooled objects; Caching infrastructure; Persistent storage or external dependency pools; Messaging infrastructure; Garbage collection.
• If a pool is sized too small then a request will be forced to wait for an object to become available. Waiting for a pooled resource increases response time, and can cause a significant performance degradation if more and more requests continue to backup waiting on the pooled resource.
• If a pool is sized too large then it may consume too much memory and negatively affect the performance of the JVM as a whole.
• Pooled objects are stateless, (it doesn't matter which instance the application obtains from the pool); Cached objects are stateful (the request for a cached object needs a specific instance).
• Because caches hold stateful objects, it is important for the cache to maintain the most frequently accessed objects in the cache and provide enough additional space in the cache for infrequently accessed objects to pass through.
• A cache must be sized large enough to minimize cache misses, but not so large as to consume too much JVM memory.
• External resource pools, such as database connection pools, must be sized large enough so that requests are not forced to wait for a connection to become available in the pool, but not so large that the application saturates the external resource.
• One of the single biggest performance improvements that can be made to a JVM is to optimize its garbage collection behavior.
• Tune backwards: Configure so that external outgoing load is maximized (allow too much load to pass through); Generate representative service requests; Identify the external reources that are saturated & configure the application to allow only enough load to pass to them so they don't saturate; Tune all other wait-points to load the server without causing requests to wait; Allow all other requests to wait at a throttle point, such as at the web server.
• Aim to send the amount of load to external dependencies that maximize their usage without causing saturation.
• All technology wait-points, such as object pools, caches, and garbage collection, should be tuned to minimize delays.

http://java.sun.com/developer/technicalArticles/Interviews/community/pepperdine_qa.html
Java Performance Tuning: A Conversation With Java Champion Kirk Pepperdine (Page last updated July 2008, Added 2008-11-27, Author Janice J. Heiss, Publisher Sun). Tips:

• Measure, don't guess. Developers are trained to look at code, and when something goes wrong, they look at code - and you can always find something wrong or ugly that's begging to be fixed. And that can throw you right off the track of the actual issue. Delay looking at the code until you have a solid measurement or clue as to exactly which part of the application is responsible for the problem.
• The box has four layers: People; Application; JVM; Hardware. Each layer is important to the overall performance of the system. If you change anything in any layer, you will change the performance profile. If you don't account for all of the layers, you run the risk of hiding performance bottlenecks or creating artificial ones. This misstep alone often results in teams wasting a lot of time.
• Complex code tends to confuse optimizing tools, so that they provide either suboptimal optimizations or no optimizations at all. Dumb code is better (and easier to read)
• If the code is well structured and loosely coupled, if the classes are cohesive, and the code uses delegation, changes needed for tuning are more localised and cuase fewer problems.
• Performance tuning often requires refactoring code changes, and all of the arguments that the Agile crowd put forth - loose coupling, simple code, following good design patterns, having good unit testing in place, automated builds, and so on - make performance tuning easier.
• If you can null out an object with no ill effects, the object is improperly scoped or scoped too broadly. A better solution would be to narrow the scope of the object so that it goes away when the value is no longer needed.
• Performance problems exhibit themselves in a live, fully functional system, so it only makes sense to look for them in a live, fully functional system. Not reading code (until you know exactly where to look).
• Ignore the code when first confronted with a performance problem. Measure, find the bottleneck, narrow down the search space, then, finally, look at the code that matters.
• Most Java to database performance problems can be traced back to overutilization or poor structure. That includes too many table joins or lack of indexes - simple things like that which DBAs can easily identify and fix. The more difficult Java to database performance problems are from too complex a communication platform that causes overly complex or too many communications for the function point.
• Fours common database issues are: JDBC logic scattered too widely to be tackled in easily; using the database for inter-process communication rather than as a data store; putting key business logic in the database; having denormalised data flow into the database.
• The first step in recognizing database overutilization is to count the number of interactions between the application and the database and compare that to the amount of work that you're doing.
• A quick fix to database overutilization is to add caching, as it's much easier to do this than eliminate the excessive calls to the database.
• Bulk up on the database interactions.
• Memory leaks are easiest to solve using the generations feature (found in the NetBeans profiler and VisualVM).
• Playing with heap-space sizings can make a huge difference in GC efficiency, application response times, and throughput.

Jack Shirazi Back to newsletter 096 contents