The Java Application Monitor (JAMon) is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor production applications. JAMon can be used to determine application performance bottlenecks, user/application interactions, and application scalability. JAMon gathers summary statistics such as hits, execution times (total, average, minimum, maximum, standard deviation), and simultaneous application requests. JAMon statistics are displayed in the sortable JAMon report. Click JAMon report to see a live demo of JAMon.

JAMon was developed primarily for monitoring J2EE applications, however it can be used in any JDK 1.2 or higher environment. JAMon can be used in Servlets, JSP's, EJB's and Java Beans in various J2EE Application Servers (Sybase's EAServer, and BEA's WebLogic,?), and can also be used in other programming environments that can call Java code (ColdFusion, PowerBuilder, BroadVision, ...).

Using JAMon is easy. Place JAMon.jar (90K) in your classpath, and surround the code that you wish to monitor with the JAMon API's start() and stop() methods. See the code sample below:

import com.jamonapi.*;
...
Monitor mon=MonitorFactory.start("myFirstMonitor");
...Code Being Timed...
mon.stop();


For example, to collect statistics for all JSP pages in a Web Application, a developer would place a call to the start() method at the top of every JSP, and a call to the stop() method at the bottom. With a few minutes work the developer will have statistics for all application pages. The statistics will include number of page hits, average page execution times, and the number of users simultaneously accessing each page. The statistics can be displayed as an HTML table by calling the getReport() method, or using JAMonAdmin.jsp.

JAMon requires no special administration accounts. Many monitoring tools require administrative accounts to install monitoring software, and to view statistics. Due to the complexity of these tools, developers typically go through specially trained administrators to access monitoring statistics. Many of these monitoring tools also require holes punched in firewalls.

JAMon has none of these problems. Statistics gathered via JAMon are available to all developers via JAMonAdmin.jsp. No administrative accounts, rights, or set-up are required for gathering and viewing statistics. Because the JAMon API is part of the application code, developers can switch J2EE App Servers and monitoring will continue to work. JAMonAdmin.jsp is accessed via HTTP and so can be viewed without punching any additional holes in firewalls.

JAMon can be used to monitor production systems. Monitoring production systems is important because it is the only way we can see the frequency that our code is being executed by real users. Also, many times the configuration of the production environment is different from the test environment, so the performance numbers between the two environments may be considerably different. For both these reasons monitoring only the test environment is not enough.

Unlike many monitoring tools, JAMon, due to its minimal performance overhead and multi-threaded capabilities, can be used in production systems. JAMon can be disabled at runtime (using JAMonAdmin.jsp), further reducing performance overhead. On a 2 Ghz Pentium IV, JAMon's start(), and stop() methods execute 500,000 times per second. After disabling, the methods execute 30,000,000 times per second.

To test JAMon's performance download JAMon.jar and execute the following command. The number represents how many times start(), and stop() should be called in the various tests (100,000 will be used as the default if a number is not provided).

java -cp JAMon.jar com.jamonapi.TestClassPerformance 500000

The last label ("Full Factory TimingMonitor()") in the output indicates how long it takes to run when monitoring is enabled, and the label starting with "NullMonitor2()" indicates how long it takes to run when monitoring is disabled.

JAMon is flexible. Many monitoring tools specialize in monitoring specific parts of an application. This can force a developer to resort to using multiple monitoring tools to cover all of their monitoring needs. For example, a developer may have to use different tools to monitor each of the following: page hits, EJB invocations, method calls and database connections,... .

JAMon's start() method takes a string as an argument. This string can be a JSP page name, an EJB name, a method name or any other code or concept that can be represented by a string.

Also, because strings are passed to the start() method, a developer can decide what to monitor based on runtime data. Using this technique we can monitor trends as well as performance bottlenecks. Some examples of JAMon's flexibility:

• Keep statistics for a JSP home page
• Monitor mon=MonitorFactory.start("HomePage");
• Keep statistics for a DataAccess class's openConnection() method (which opens a database connection)
• Monitor mon=MonitorFactory.start("DataAccess.openConnection()");
• Keep statistics for each user that logs into our system
• Monitor mon=MonitorFactory.start(getUserName());
• Keep statistics for all page accesses by date. For example, "pageAccesses.05/10/02"
• Monitor mon=MonitorFactory.start("pageAccesses."+getTodaysDate());

JAMon gathers statistics for any code that comes between start() and stop() methods. In our first somewhat artificial example we will be timing how long our program sleeps when calling the Java Thread.sleep() method ten times from within a "for" loop.

import com.jamonapi.*;


public class MonitorTest {
    public static void main(String[] args) throws Exception {
        Monitor mon=null;
        for (int i=1; i<=10; i++) {
            mon = MonitorFactory.start("myFirstMonitor");
            Thread.sleep(100+i);
            mon.stop();
        }
        System.out.println(mon);  // toString() method called
    }
}

MonitorFactory.start("myFirstMonitor") both creates a monitor and begins gathering monitoring statistics. The start() method takes a monitor name/label as an argument. Summary statistics are gathered for all monitors that are passed identical labels. In our example the start() method is called 10 times with the same label, so the summary statistics will include data from all 10 calls.

The next line in the example is the sleep() method. The sleep() method takes one argument, which is the number of milliseconds the active thread should sleep. The loop has the effect of sleeping the thread for 101,102,?,and 110 ms. Finally comes the stop() method which will stop gathering statistics.

After the loop we print the statistics to the console, taking advantage of the monitor's toString() method. Let's take a look at what these statistics mean:

110 ms. - The execution time (in milliseconds) of the last monitor that was stopped. (There are 1,000 milliseconds in one second.)

The rest of the line contains summary statistics for the ten monitor calls. Considering our logic timed the sleep method 10 times with values ranging from 101 to 110 ms. the following results seem reasonable.

Hits=10 - A hit occurs whenever the start() method is called with an identical label. In this case "myFirstMonitor" is the monitor label.

Avg =106 ms. - Average is the total execution time divided by hits (i.e., Total/Hits).

Total=1,062 ms. - Total represents the total accumulated time for all 10 monitors that were executed. If we add 101 through 110 we get 1,055 ms., which is close to the total time the monitor calculated. The sleep() method is approximate, so our total is not exactly 1,055 ms.

Min=100 ms. - Min is the minimum execution time for the 10 hits.

Max =111 ms. - Max is the maximum execution time for the 10 hits.

Active=0 - The "Active" statistics have meaning in a multi-threaded environment. It indicates the number of current simultaneously executing monitors with identical monitor labels. This information could be used in a JSP to tell how many users are currently executing a page. In our example all monitors have been stopped prior to calling toString(), so no monitors are active.

Avg Active=1 - Average active indicates the average number of simultaneously executing monitors with identical monitor labels. In our single threaded example we would expect this to be one.

Max Active=1 - Max active indicates the maximum number of simultaneously executing monitors with identical monitor labels.

First Access=5/2/02 10:11:48 AM - Indicates when the monitor with the given monitor label was first executed.

Last Access=5/2/02 10:11:49 AM - Indicates when the monitor with the given monitor label was last executed.

In our second example we will use JAMon to gather page statistics for a servlet. To gather page statistics a developer places a start() at the top of the servlets doGet() or doPost() method, and a stop() in the finally clause. If a method that can throw an exception is being monitored, then the stop() method must be called from the method's finally clause. This ensures that the monitor is stopped when an exception is thrown.

The sample servlet doesn't do much. It simply returns a list of musicians from three legendary reggae groups (Sample Output). The servlet sleeps for three seconds which slows down processing enough to get simultaneous servlet instances running. The getMusicians() method sleeps an amount based on the number of servlet instances currently running. This will help in demonstrating how JAMon can be used to measure application scalability. The statistics generated by this example will be discussed in the next section.

The code for the example follows:

import javax.servlet.*;
import javax.servlet.http.*;
import java.io.*;
import java.util.*;
import java.text.*;
import com.jamonapi.*;


public class JAMonDemo extends javax.servlet.http.HttpServlet {
   private static int active;
   protected void doGet(HttpServletRequest request, HttpServletResponse response)
    throws ServletException, IOException     {
        Monitor primary=MonitorFactory.startPrimary("JAMonDemo.pageHits");
        Monitor byDate=MonitorFactory.start("JAMonDemo.pageHits."+getDate());
        Monitor byUsersIP=MonitorFactory.start("JAMonDemo.pageHits."+
        getFormattedIP(request));
        active++;


        response.setContentType( "text/html" );
        PrintWriter out = response.getWriter();


        try {
            Thread.sleep(3000);
            out.println(getMusicians("Culture"));
            out.println(getMusicians("Maytals"));
            out.println(getMusicians("Wailers"));
        }
        catch(Exception e) {
            throw new ServletException(e.getMessage());
        }
        finally {
            byUsersIP.stop();
            byDate.stop();
            primary.stop();
            active--;
        }
    }


    // Returns date as a string in 05/10/02 format.
    private String getDate()     {
        return DateFormat.getDateInstance(DateFormat.SHORT).format(new Date());
    }


    /*
      Replace the standard "." separated ip address with one separated by dashes.
      i.e., 111.222.333.444 becomes 111-222-333-444.  This is done because "." has
      special meaning to the monitor class and will consume more resources.
      However this is not required.
     */
    private String getFormattedIP(HttpServletRequest request)     {
        String ip=request.getRemoteAddr();
        if (ip==null || "".equals(ip))
            return "";
        else
            return ip.replace('.', '-');
    }


    private String getMusicians(String group) throws Exception {
        Monitor mon=MonitorFactory.start("JAMonDemo.getMusicians("+group+")");
        final String culture="Joseph Hill, Albert Walker, Kenneth Dayes";
        final String maytals="Toots Hibbert, Raleigh Gordon, Jerry Matthias,"+
        " Jackie Jackson, Winston Wright, Hux Brown, Rad Bryan, "+
        "Paul Douglas, Winston Grennan";
        final String wailers="Bob Marley, Peter Tosh, Bunny Wailer, "+
        "Junior Braithwaite, Beverly Kelso";


        String musicians=null;
        if ("Culture".equals(group))
            musicians="<b>"+group+"</b>"+" - "+culture;
        else if ("Maytals".equals(group))
            musicians="<b>"+group+"</b>"+" - "+maytals;
        else if ("Wailers".equals(group))
            musicians="<b>"+group+"</b>"+" - "+wailers;


        Thread.sleep(active*50);
        mon.stop();
        return musicians;
    }
}

This example demonstrates the flexibility of the JAMon API, by incorporating runtime data such as date and IP address into our monitor labels. We are monitoring the following items in our servlet:

JAMonDemo.pageHits - Every time the JAMonDemo servlet is executed another hit will be recorded for this monitor label. This monitor is recording page hits. The consequences of starting this monitor with the startPrimary() method will be discussed later.

JAMonDemo.pageHits.Month/Day/Year - This monitor label will monitor page hits by day. There will be one row in the monitor report for each day. Typical entries would be "JAMonDemo.pageHits.5/12/02" and "JAMonDemo.pageHits.5/13/02".

JAMonDemo.pageHits.UsersIPAddress - This monitor label will monitor page hits by users IP address. There will be one row in the monitor report for each IP address. Typical entries would be "JAMonDemo.pageHits.127-0-0-1" and "JAMonDemo.pageHits.205-130-238-40". If the application has accounts then user names could be used instead of IP addresses.

JAMonDemo.getMusicians(group) - This monitor label will monitor calls to the getMusicians() method. The argument that is passed to the method is also incorporated into the monitor label. This adds little value in this example, but in a more real world program this technique can be very powerful. Incorporating class and method names into the monitor label (i.e., MyClass.myMethod) makes it easier to match monitors to the code they are monitoring.

JAMon comes with a JSP that allows developers to view the JAMon report and control JAMon behavior. JAMonAdmin.jsp can be found in the JAMon.war download.

The JAMon report has one row for each unique monitor label that was passed to the MonitorFactory.start("?") method. This row is called a "JAMon report entry". The JAMon report displays all monitor information in one HTML table. The following statistics were discussed in enough detail in the previous example and won't be covered here: monitor label, hits, average, total, minimum, maximum, first access and last access.

We will be referring to the JAMon report in the following text (note sorting will only work when viewing the JAMon report from JAMonAdmin.jsp).

Looking at "JAMonDemo.pageHits" JAMon report entry we can see that the servlet had 30 page hits taking on average 3,441 ms. Some other statistics that require further discussion follow:

Std Dev ms. - 208 ms. - This is the standard deviation for the execution time of the 30 hits. If the statistical distribution is a normal distribution then we would expect 68% of the our hits to be within one standard deviation of the average, and 95% of our hits to be within two standard deviations.

Active - 0 - Indicates the number of monitors with this label that are currently executing (i.e., "Active" indicates concurrency). In the example when the JAMon report was run, no users were waiting for the JAMonDemo servlet to return, so "Active" was 0. However, if five requests were currently being handled by the servlet then the "Active" value would be 5.

Avg Active - 2.8 - Indicates the average number of monitors with this label that were simultaneously executing. In this example, of the 30 page hits, there were on average 2.8 simultaneously executing pages.

Max Active - 5 - Indicates the maximum number of monitors with this label that were simultaneously executing. In this example, of the 30 page hits, the most that were executing simultaneously was 5.

Note the JAMon report in JAMonAdmin.jsp can be sorted by clicking column headers. Clicking will toggle between ascending and descending sorts, based on column values. An up arrow indicates ascending, and a down arrow indicates descending.

The JAMon report can be displayed in a number of different ways. Two of the ways follow:

MonitorFactory.getReport() - Returns the JAMon report as an HTML table in alphabetical order by monitor label.

MonitorFactory.getRootMonitor().getReport(int sortCol, String sortOrder) - Returns the report formatted as an HTML table. The method is passed the column number to sort by, and whether the sort should be in ascending ("asc") or descending ("desc") order.

A problem with the statistics previously mentioned is that they hide extreme data points (i.e., outliers). When application performance degrades we become more interested in outliers. JAMon time ranges contain information about these outliers.

Ideally we would like the average time for a monitor to be close to its minimum. However many factors (such as number of simultaneous users) can cause performance degradation. For example, we would expect our servlet to respond faster with two simultaneous requests than with 5,000.

There are 13 JAMon time ranges. The ranges are measured on an exponential scale. The first range is from 0 to 10 ms., and subsequent ranges double, until the final range, which includes values greater than 20,480 ms. We will use the JAMon report entry "JAMonDemo.getMusicians(Wailers)" and its "81-160ms." time range to discuss the five time range variables. The time range values are "7/114 (1/2.3/7.9)".

Hits - The number of hits within the given time range. All time range hits sum to the total hits for the JAMon report entry. The value for hits in the example is 7. This means that of this report entries 30 hits, 7 of them fell in the "81-160ms." range and averaged 114 ms.

Average - The average time in ms. for all hits within the given range. (see Hits for further info).

(Avg Active/Primary Active/Global Active) - All three of the time range "Active" statistics provide information about application scalability, by correlating application performance to monitors that are currently running. When a monitor is started the various active values are increased by one (based on the rules specified below) and when a monitor is stopped the active values are decremented by one. In general we expect performance to degrade with increasing activity. The time range "Active" statistics are all averages.

Avg Active - Indicates how many monitors with this label were on average "Active" when a hit occurred within the given time range. This allows us to correlate performance of this monitor to itself. This is similar to the "Avg Active" entry that comes earlier in the report. The difference is that this average is only for the hits that occur within this time range. In our example for the 7 hits within this time range on average only one monitor with the label "JAMonDemo.getMusicians(Wailers)" was executing concurrently.

Primary Active - Indicates how many primary monitors were "Active" on average, when a hit occurred within the given time range. A primary monitor is simply a monitor started with the startPrimary() method instead of the start() method (See the second example). This feature allows a developer to correlate the performance of all of their different monitors to a primary monitor. Primary monitors should be chosen to correlate highly with performance.

Page hits is a good choice for a primary monitor. We would expect as the number of simultaneous page hits increases performance will decrease. In the example we used page hits as our primary monitor ( MonitorFactory.startPrimary("JAMonDemo.pageHits") ).

In our example there were an average of 2.3 pages executing when the monitor labeled "JAMonDemo.getMusicians(Wailers)" had an execution time in the "81-160ms." range. The time ranges indicate that this method may have problems scaling. Notice how performance is decreasing with an increased number of active pages.

41-80ms. - When hits were within this time range there was an average of 1 page active.

81-160ms. - When hits were within this time range there was an average of 2.3 pages active.

161-320ms. - When hits were within this time range there was an average of 4.1 pages active.

"Avg Active" and "Primary Active" will be the same for primary monitors. If a primary monitor is not chosen then "Primary Active" will be 0.

Global Active - Indicates how many total monitors were "Active" on average, when a hit occurred within the given time range. The global active counter is incremented whenever a start() or startPrimary() method is executed (regardless of the monitor label), and decremented whenever the stop() method is executed. This feature allows a developer to correlate performance to the number of things being monitored in the application.

Notice how performance is decreasing with an increased number of global active monitors:

41-80ms. - When hits were within this time range there was an average of 4 monitors active.

81-160ms. - When hits were within this time range there was an average of 7.9 monitors active.

161-320ms. - When hits were within this time range there was an average of 13.4 monitors active.

Measuring "Primary Active" has an advantage over "Global Active". If we start monitoring more things then "Global Active" will artificially increase. In the above example if we decided to monitor "JAMonDemo.pageHits.UsersIPAddress.Month/Day/Year" then all of the "Global Active" variables discussed above would increase even though there is really no more activity on the server. This is not true of "Primary Active".

To summarize the concept of "Active" monitors let's look at another example. If the following code were executed there would be six active global monitors, two active primary monitors, two active monitors for "mon1", and one active monitor for all of the other monitors.

MonitorFactory.start("mon1"); // 1 global

MonitorFactory.start("mon2"); // 2 global

MonitorFactory.start("mon3"); // 3 global

MonitorFactory.startPrimary("mon4"); // 4 global, 1 primary

MonitorFactory.start("mon1"); // 5 global, 1 primary, and 2 mon1's are active

MonitorFactory.startPrimary("mon5"); // 6 global, 2 primary

The JAMonAdmin.jsp page also allows a developer to control JAMon at runtime. A description of the function of each of the buttons follows:

Refresh - Redisplays JAMonAdmin.jsp with the JAMon report in the default sort order. Refreshing can also be accomplished by clicking a column header.

Reset - Clears all monitors and their statistics. The report will now be empty, and subsequent calls to the start() method will repopulate JAMon statistics.

Enable and Disable - JAMon can be enabled and disabled at runtime. The default is enabled. When the disabled button is pressed JAMon statistics will no longer be gathered, however existing statistics will not be reset. To begin gathering statistics again simply press the enabled button.

This section discusses the various ways JAMon can be used. JAMon's most obvious use is in determining application performance bottlenecks by gathering timing statistics. However, it can also be used in other ways such as determining how users interact with an application, and determining application scalability. Some ways that JAMon can be used follow.

To tune performance - Web applications can be difficult to tune due to the fact that there are many possible bottlenecks including: network IO, database IO, application code, poorly tuned queries, or even an increased amount of site traffic. To help pinpoint problems, performance diagnostics must be built into the application.

Without performance measurements, developers will spend their time guessing where performance bottlenecks occur. Performance problems typically occur in a small percentage of overall code. Algorithm guru Donald Knuth estimated that 4% of application code accounts for 50% of the performance. With measurements, developers can quickly locate that 4% of the code and get the biggest bang for their tuning buck. Without measurements developers waste effort tuning code that may have no significant impact on performance.

To improve code design - "More computing sins are committed in the name of efficiency than for any other single reason-including blind stupidity" - W.A. Wulf

"...premature optimization is the root of all evil." - Donald Knuth

Often, when discussing code designs, developers say they don't want to implement certain elegant designs, because of perceived performance problems. JAMon supports the following argument: "Let's code the elegant solution and measure its performance. If it doesn't perform well then we can try the other design."

To determine how users interact with the application - JAMon can be used to answer questions of the following type:

• What is the application's peak usage time?
• What is the maximum number of simultaneous users for the application?
• Who are the application's "power users"?
• Are certain application pages never used?
• When did a specific user last login to the application?
• What are the most commonly searched product categories?

To set coding priorities - Coding priorities can be based on factors such as which features are accessed the most often or what code is the slowest.

To detect programming and user errors - JAMon can be used to track errors, such as when exceptions are thrown, when resources are not released or when a user error occurs such as forgetting a password.

Developers can monitor code that gets and releases a resource, and check to see that the number of hits for both are the same. For example, if we get 1,000 database connections, we should also release 1,000 database connections. MonitorFactory.start("getConnection") would be called when a developer gets a connection and MonitorFactory.start("releaseConnection") would be called when the connection is released. If the hits don't match then we have a resource leak.

Developers can also monitor when Exceptions are thrown or when assertions are triggered. One helpful way to track Exceptions would be by date by user (i.e., "Exception.IllegalArgumentException.05/15/02.jsmith"). This allows developers to determine when exceptions are thrown without depending on users for the information.

To test application scalability - JAMon time ranges correlate performance to application load. Using this information a developer can measure how well an application scales. See the description of JAMon time ranges for more information on how to use JAMon for scalability testing.

To determine what is happening in an application at a snapshot in time - Many monitoring tools monitor an application from the "outside looking in", but because JAMon looks at the application from the inside we can use statistics such as "Active", "First Access" and "Last Access" to determine what activities our application is performing at any given time.

Currently JAMon does not persist its statistics. When the JVM goes down the statisistics are lost. A future version will be able to persist monitoring statistics to a database. This will allow developers to track how application usage and performance evolves.

Persisting statistics will also allow developers to run monitoring reports that span all tiers of an n-tier application. For example, if we have a clustered application using three web servers, three application servers and one database, we could persist statistics from all these servers into one database. This would allow us to have one report that shows application monitoring statistics for all tiers.

Java code can be called from both Java and non-Java environments. JAMon can be used in any environment that can call JDK 1.2 code. JAMon has been successfully used in ColdFusion script, BroadVision JavaScript and PowerBuilder. JAMon should also work in Active Server Pages, Java in the database, and many other environments. Sample code follows:

Servlet Filters - A Servlet filters is a simple piece of Java code that is executed whenever a Java Web Application resource is accessed. Resources include Servlets, JSP's, GIFs, JPeg's, and HTML documents. The simple filter example will monitor all of these. By using JAMon the filter becomes a powerful web site hit counter.

HTTP Sessions - The servlet API provides an interface that allows developers to track when an object is bound to an HttpSession. If the object implements the HttpSessionBindingListener interface then the valueBound(...) method is called when the object is put into the session, and valueUnbound(...) is called when the object is removed from the session. This allows an application to use JAMon to monitor statistics related to HTTP Sessions such as how many sessions are active, and which users are currently using the application.

Monitoring PowerBuilder Code - The sample code contains a Monitor EJB that acts as a thin wrapper for JAMon. The PowerBuilder client takes a monitor label as input and has a start and stop button. The start button passes the monitor label to the EJB and the stop button stops the Monitor. JAMon.jar must be accessible to the EJB at runtime.

ColdFusion sample

BroadVision sample

I will continue to add sample code to this document.

Click here to download JAMon jar, which contains the JAMon API (approximately 90K).

Click here to download all of JAMon (approximately 225K). This download contains:

• JAMon.jar - Contains the JAMon binary (same as download above)
• JAMon.war - A sample Java Web Application that uses JAMon. The WAR can be installed in any J2EE compliant application server. The WAR contains a servlet that generates monitoring statistics (/JAMon/JAMonDemo), and a JSP that allows us to look at the statistics (/JAMon/JAMonAdmin.jsp).
• JAMon Java Docs
• JAMon Source Code - The full source code used to build JAMon.jar. There are approximately 1000 lines of code in JAMon, and it has some nice examples of the Gang of 4 design patterns including the creational, iterator, command and decorator patterns.

JAMon has a very liberal license agreement. JAMon binaries can be used free of charge, and JAMon source code may be modified. The JAMon License was adapted from the BSD license. It is requested that any modifications developers make to JAMon be sent to admin@jamonapi.com , so that all JAMon users may benefit.

Steve Souza is a senior consultant working for Sybase Professional Services in support of the General Services Administration (GSA) in Arlington, VA. He has 20 years of software and database development experience. Please send any questions or comments concerning JAMon to admin@jamonapi.com.

I'd like to thank Ed Desrosiers for his help in designing JAMon, and Eric Laufer for his help in designing the JAMon report and converting this article to HTML.