There are two RootCause memory probes: memstat and java_memstat.
The two are similar in function: to help you understand the memory usage of your application and specifically, to help find why an application grows in its memory usage over time. java_memstat tracks Java objects, while memstat tracks allocations by native languages such as C/C++.
The two probes can be run on production applications; no special debug build is needed. Each of them produces a summary report that states whether memory growth is occuring and how bad it is. This report is simple to understand. Each also produces detailed reports that are appropriate for software engineers to diagnose and fix the growth problems, if the summary reports says that such problems exist.
Since the two are so similar, they are documented together.
This section is intended to be used for a quick start to the memory probes. Subsequent sections discuss the memory probe features in more detail.
If you're new to RootCause, run through the Pi demo first, as described in Chapter 5 of the RootCause User Guide. There is a Java version and a C/C++ version of this small demo program included with RootCause, and it will give you basic familiarity with the RootCause product. Don't worry too much about the tracing that is done in the Pi demo, as the memory tracking probe does not use that functionality, just concentrate on making the workspace and examining the resulting data.
Now we will set up to run the memory probes on your application (whether it is J2SE, J2EE, C/C++ or a third party product). RootCause can deploy probes to a remote site, but in this quick start, we will assume that you will run the RootCause console on the same computer as the application.
First, we will make a workspace for your application, just like was done in the PI demo.
Turn RootCause on. Run your application normally and terminate it (if it is an app-server, you can just stop the app server). Now bring up the RootCause console (on windows) or execute the "rootcause open" command (all other operating systems). This will bring up the rootcause log and the main workspace window.. You will find your application in the rootcause log. Normally, you will recognize your application immediately, but if you are running an app-server, it might be a bit more tricky, since the JVM may be embedded. Here are some hints on some popular app-servers:
In the log, highlight the application that you wish to monitor and then use the right mouse button to open a menu and choose "Open associated workspace". This will open a pane with some options. Accept the defaults, except for perhaps the location of the workspace. (The workspace is a directory where RootCause puts all its files). Set the workspace location option to be a local disk with "enough" space (maybe 100M just to be totally safe). Click OK on the create workspace dialog and then click OK on the register application dialog.
Now, you can close the rootcause log window and there will still be the main workspace window open. It will have your application listed in the window. There will also be a few checkboxes underneath the application. This is the main RootCause window and you can do many things from here, but all we will do is use the memory probes.
Second, we will tell RootCause to apply only the memory tracking probes to your application.
Uncheck all the boxes except for java_memstat (if you want to track Java objects) and memstat (if you want to trace native code leaks) or both. Don't check java_memstat if you do not have a java application!
Now click the Build button along the top of the window. After that completes successfully, you are ready to run the application again. When the build finishes, you should see something like the following:
Third, we will run the application with the memory tracking probes.
Run the application normally. RootCause will intercept the creation of the new process and add the memory probes to the memory image of the process, then let the application run. The memory probes are collecting data as the application runs. Run it for long enough to collect some interesting data.
Note that the default parameterization assumes the application will run for at least half an hour. See A Note About Parameterization below. The first time through you needn't worry about this -- the idea is to get a feel for the process and what the results look like. Then you can start gathering serious data:
Exercise the application in a normal environment with a representative load so that you will collect data that is representative of the real load on the application.
Fourth, we will examine the collected data by formatting it.
In the RootCause Console, click the Examine button as shown below.
A window will appear that has a number of data files listed for the process that RootCause just intercepted. (If you have run the application more than once, there may be data files for more than one process. In this case, only look at the data for the most recent process.) Since you want to use all of the collected data for the memory analysis, select all of the data files for that process and then click OK.
An event window and a graph will then come up (see following screen capture). If you both the memory analysis probes, then two graphs will appear.
The graph shows the overall memory action (allocations, deallocations and outstanding). You can use ctrl-click to zoom into portions of the graph and shift-click to move the graph (the reset button will put the graph back to it's original look).
In the Trace Events window, you will see an entry for each of the memory probes that you had selected before you ran the application. Select one of these and click MB3 (right mouse button) to bring up a menu and select "Show Associated Table". The resulting table will tell you where the textual reports are for the memory probes. Use your favorite text editor to look at these text files. You will want to look at the summary report first. It summarizes the growth observed by the RootCause memory probes.
Fifth (optional), we will reformat the data one or more times to get different views of the collected data. In step 4 above, we formatted the data for the entire program into a set of reports. By changing some of the format parameters for the memory probes, we can have create graphs of outstanding allocations and perform analysis on specific time segments rather than the entire program. The idea here is that you start with the whole program and look at the collected data and then you reformat to examine specific types and time slices. (See Format Time Options.)
The memory probes have a default parameterization that should work well for J2EE/J2SE/C/C++ applications that run with a representative load for a reasonable amount of time and (if Java) do garbage collections on a fairly regular basis. Here are some discussions about the default parameterizations and what you may want to change.
That's the Quick Start part. If you haven't gotten some useful information yet, contact OC Systems support. The following is more detailed information about usage of these probes.
The two memory probes may be run simultaneously, so that a Java application that has C++ JNI calls can have both kinds of growth tracked and debugged at the same time. These probes may also be applied to third party products. For example, if you run both probes on a J2EE application, the probes will detect memory leaks for Java Objects, JNI code, J2EE support code, the JVM itself, as well as any third party products used by your application!
As the application is run under RootCause, the java_memstat and memstat probes collect and log data efficiently in a raw format to binary files within the RootCause workspace.This data is then formatted, (maybe even on a different machine if you have deployed the memory probes). When this data is formatted, the probes produce reports detailing whether the application is growing over time and where that growth occurs.
Note that the memory probes are standard RootCause probes, written using the RootCause custom probes feature and integrated into RootCause using the standard plug-in mechanism. The RootCause intercept mechanism is used, so no application files need to be modified to use the memory probes.
The probes are configurable. The default settings should be fine for an application that is run for more than half an hour with a production type of load. No matter what, it should only take a few minutes to set up RootCause to run the memory probes on your application.
In a typical usage of the memory probes, one would collect the data
from the application once and then (in an offline manner) format the
data one or more times, with different format options each time
to get different views of the collected data. Each time the collected
data is formatted, a new set of reports are generated.
There is a separate set of reports for the Java and native probes.
Terminology
The memory probes operate on both Java and native languages. Both Java and native language application can suffer from "memory growth" but the technical reasons are different. Java programs accumulate objects that are no longer needed, but are still referenced, native language applications accumulate memory when the application does not explicitly release the allocated memory.
The two RootCause memory probes are purposely very similar to each other. To simplify this discussion, we will will refer to both object creation (in Java) and memory allocations (e.g., the "new" operator in C/C++) as "allocations" and object deletion (i.e., garbage collection in Java) and the memory free operations (e.g., the free operator in C/C++) as deallocations.
The memory probes collect statistics based on an "allocation id". This id is represented by a specific traceback for native languages (such as C/C++) and by a traceback and a type for Java programs. One can search the reports for the allocation id to find information regarding the allocation point.
An application that grows over time will eventually exceed the memory limits and fail. Applications normally have swings in memory size, in fact, some applications will have large swings in the memory usage and never fail; they grow for a period of time and then shrink back down at a later time. Because of this, short runs of an application are not good for debugging growth problems or even knowing if a growth problem exists. In the general case, an application must run for a long period of time to be certain that there is truly a growth problem (or not).
The RootCause memory tracking probes are designed to be able to run for a long period of time so that the expected memory usage swings "average out". They are designed to be used while stressing the system, perhaps by a load testing scenario or by using them in production where the application will stay up for a long period of time.
The general idea with the memory probes is "the longer the run, the better". You can look at the data during the run, without taking the application down.
Using the memory probes is straightforward:
The memory probes are quite configurable. As stated previously, the default settings are probably fine for an application that will run for at least half an hour, but you should probably read this section to get a feel for how the probes work.
There are two memory probes: one for Java and one for C/C++. They are independent of each other. Each is parameterized separately from the other, but the kinds of parameterization for each are very similar, so the two are discussed at the same time.
There are two sets of options for each of the two memory probes: runtime and format time. The runtime parameters affect how the data is collected. The format time parameters define how the collected data is processed and presented to the user.
The parameterization of the memory probes is accomplished by double clicking on the memory probes in the workspace window of the RootCause Console. A new window will appear that has a separate pane for the Runtime and Format time options. A pane will also appear for Advanced options. You will not need to set any advanced options except at the suggestion of support@ocsystems.com
The following screen grab shows the runtime paramaterization pane of the memory probes. To bring up this pane, double click on the memory probe in the workspace window.
The interval is specified as number of seconds. For example, an interval of 60 means that the garbage collector will run at least once every 60 seconds.
How do you know how big to make it? To start with, there is no real disadvantage to setting this limit large. Just make sure that you have enough disk space; we have seen users collect a gigabyte before. How can you tell if older data has been discarded for a particular run (and therefore you might want to make the limit larger for a subsequent run)? In the RootCause console, click Examine. It will bring up a dialog that will show you the series of files where RootCause logs the data. The are numbered sequentially. If you do not see a file with the sequential number 1 in it (e.g., Pi.class-1.apd) then that file has been deleted and you might want to increase the limit as above. Both memory probes use the same logging mechanism; also, this parameterization is not specific to the memory probes. When you increase/reduce the log file size, all probes are affected, including the memory probes.
The following screen shot shows the format time options of the memory probes. To see this pane, double click on the memory probe in the workspace window.
The memory probes collect data during the running of the application. At format time, this raw data is used to create meaningful data to determine why an application is growing. It is likely that you will run the format a number of times to do different analysis's of the data.
TestAllocs.aws/TestAllocs.class.java_memstat.cfg.
As suggested, these are for those who have a lot of experience
with the probes, primary for our own consultants.
The configuration file contents for memstat and java_memstat
are are described in $APROBE/probes/memstat.cfg and
$APROBE/probes/java_memstat.cfg, respectively.
Report Files
The memstat probes produce a number of reports and associated data. Exactly which reports are produced depends on the options you select at format time (see above) and whether the relevant data was recorded at runtime. By default the reports go into a number of files in the workspace directory; you can change the directory and include the process ID in the filename if you choose.
There are four possible files produced by each of the memstat and java_memstat probes. The default names of these files is based on the name of the application and the name of the probe:
application.java_memstat.traceback.txt
If you have selected a Detailed Report you will also get a list of every traceback recorded at runtime. In this case you will see the same point twice for each entry that appears in the summary.
application.java_memstat.details.txt
The summary analysis summarizes the formatted data. It lists some specific metrics and provides a statistical and heuristic analysis of where the growth is occurring. The following is a screen grab of a the analysis report:
The memory probes are designed to identify where the application growth occurs. At format time, the probes perform analysis on the collected data and try to identify the leaking areas in descending order of importance. To do this, the probes use various heuristics and statistics described herein. Note that the other reports can be used to view various aspects of memory usage for the application; the Analysis report is intended to identify the growth areas in the application.
(Note that we are just discussing the Analysis portion of the reports here. All of the other data collected by the memory probes is "factual" in that the actual collected data is reported. The Analysis interprets and summarizes the collected data.)
The memory probes collect the allocations and deallocations for various allocation ids into specific time segments for ease of examination and this table is shown as the Interval Report. This report shows the statistics for the various allocation ids grouped together into time slices. One can examine this table to see the exact statistics over the periods of time for the formatted data. In this manner, one can see the changes over periods of time for each allocation id. One can also view this data graphically by putting the allocation id in the Plot Allocation Id box and reformatting the data.
The following screen grab shows an interval report.
The graph(s) of overall growth (for both object growth and memory
leaks) provides a good overall view of the application's memory
behavoir. You can use the
In the quick start guide above, we chose to format all of the collected data. This is the best way to determine overall growth in the application. One can format smaller time segments of the data, however. There are two ways of doing this. The first is to only choose some of the data files in the examine dialog. This gives a coarse manner to view a time slice.
A more precise manner is to set the format options to a specific start time and stop time. When the format is rerun, the probes will ignore data collected outside those times. Typically, then one would view the summary graph(s) and see time segments where the memory activity was interesting, then set the format time parameters on the probe(s) and rerun the format to "focus" in on those areas. Note that the summary text reports are overwritten each time you re-format the data, so you must copy them elsewhere if you want to preserve them
The text summary reports identify each allocation by an integer id preceded by a "@". If you want a graph for specific allocation ids, then you can place those ids in the format parameters (there is a box for them, one per line), build and re-format the data and a graph will appear for each new allocation id.
So far, we have discussed running the memory probe(s) alone, without any other probes. The memory probe(s) can be run with other probes too. For example, you can trace a specific method entry or exit (as is done in the Pi demo) and run with the memory probes at the same time. When you format the data, you will see the the method entries and exits in the trace events display as well as the data normally collected by the memory probes.
You can then use the times for the method entries and exit to select time segments to view the collected data. In this manner, you can correlate specific actions or transactions to memory activity.
The output from a Java object dump is written to a self-contained file which
has everything known about the state of the objects at the given point in time.
Note that each object dump will overwrite the file so you will be left with
the last one found. If you wish to view an intermediate file, use the start
and stop time format-time options to select a
smaller slice of the data.
There are three parts to the formatted data:
Object Dump Contents
The object dump output is a fairly raw representation of the state of the Java heap as returned from the JVM itself. For each element returned to us, we indicate what the type of element was and the objects (if any) that it references. For some elements we will also display the objects (if any) that referred to it.
The following lists each elements (with an example):
#10 Unknown Root 0xdad22e08
#111 Jni Global Root 0xdaccd268
#96 Jni Local Root 0xdaccd450, Traceback (from frame -1):
java.lang.Thread::sleep(long)
==> TestAllocs::main(java.lang.String[])
#30 Java Frame Root 0xdad22e08, Traceback (from frame 0):
java.lang.Object::wait(long)
==> java.lang.ref.ReferenceQueue::remove(long)
==> java.lang.ref.ReferenceQueue::remove()
==> java.lang.ref.Finalizer$FinalizerThread::run() @ 174 (Finalizer.java)
#64 Native Stack Root 0xdaccd7a0, Traceback (from frame -1):
java.lang.Object::wait(long)
==> java.lang.Object::wait()
==> java.lang.ref.Reference$ReferenceHandler::run() @ 114 (Reference.java)
#93 Thread Block Reference 0xdad22d10, Traceback (from frame -1):
java.lang.Object::wait(long)
==> java.lang.Object::wait()
==> java.lang.ref.Reference$ReferenceHandler::run() @ 114 (Reference.java)
#123 Java Class 0x0035fd0c: java.lang.String
References to:
#131 0xdaccbaf8: java.io.ObjectStreamField[]
#7920 0xdaccbd58: java.lang.String$CaseInsensitiveComparator
#8193 Object Instance 0xdacdf1f8: Class 0x00508ca4: java.net.URL
References to:
#8153 0xdacdde68: java.lang.String
#8111 0xdacd10d8: java.lang.String
#8192 0xdacdf1e8: java.lang.String
#8158 0xdacddfc0: sun.net.www.protocol.file.Handler
References from:
Object Instance #8490 0xdace9130: sun.misc.URLClassPath$FileLoader
Object Instance #8497 0xdace91c0: java.util.HashMap$Entry
Object Instance #10557 0xdad22e20: java.security.CodeSource
Object Array #455 0xdacdf888: java.lang.Object
#455 Object Array 0xdacdf888: Class 0x0035904c: java.lang.Object
References to:
#8193 0xdacdf1f8: java.net.URL
#8196 0xdacdf280: java.net.URL
#8201 0xdacdf338: java.net.URL
References from:
Object Instance #8212 0xdacdf878: java.util.ArrayList
#4096 Primitive Array 0xdc4a4958: int[], number of elements 1000
References from:
Object Instance #11876 0xdc4a97c0: java.util.LinkedList$Entry
#9751 Object Instance 0xdb04eb80: Class 0x00549524: java.util.LinkedList$Entry
Alloc Point @15
References to:
#1965 0xdb04bc88: int[]
#9750 0xdb050b30: java.util.LinkedList$Entry
#9763 0xdb03b150: java.util.LinkedList$Entry
References from:
Object Instance #9750 0xdb050b30: java.util.LinkedList$Entry
Object Instance #9763 0xdb03b150: java.util.LinkedList$Entry
There are many Sun JVMs available. At the time of writing there
are over 40 different JRE downloads available at http://java.sun.com,
each for three different platforms and many containing multiple JVMs!
As might be expected, some of these have bugs in their implementation
of the JVMPI interface used to implement the java_memstat functionality.
We attempt to work around as many as these as
we can and, in some cases, automatically drop functionality if we
determine a specific JVM is used.
The following sections summarize issues you might find with different JVMs.
If you find problems with specific JVMs beyond those mentioned, please
contact support@ocsystems.com.
Java 1.2.2
The only real issue with Java 1.2.2 occurs on Solaris where there are two flavors of each release: The reference implementation and the production release. The production release cannot be used by RootCause's java memstat probe because it does not support the JVMPI interface necessary. If you attempt to use such an interface you will get the following error:
(W) The JVM you are running does not support the JVMPI interface version 1.
If you are running Java version 1.2 you must use the reference
implementation because the production implementation does not have JVMPI
support. You can use the java -version command to see the version you
are running. A production version will report itself as 1.2.2_x and the
reference implementation will report itself as 1.2.2_0x. For instance
1.2.2_06 (production), 1.2.2_006 (reference).
If you are unsure as to why this JVM is not supported, please contact
support@ocsystems.com.
The warning details how to find out which JVM this is. Note that this
limitation does not apply to the trace probe or to custom Java probes
that you might write: Those probes will work on the reference or the
production versions of the 1.2 JVM.
This was the first release of the hotspot JVM and has a rather fragile
JVMPI interface. On Windows we strongly recommend that you do not use
a 1.3.0 version of the JVM, it may run but we have seen many crashes
which are readily reproducible with the built-in hprof. On Unix it
mostly works OK but we have seen some errors (again reproducible with
hprof). Note that on Windows this version will not allow you to take
an object dump when the program is terminating so that option is
ignored (with a warning).
Java 1.3.1
This has a much more stable JVMPI interface than 1.3.0. There is one
issue (also present in the 1.3.0 JVMs): The JVM will crash if we
request information about certain unknown classes. Since the
determination about whether we can safely request the class or not
cannot be known without knowing what the class is (an obvious
Catch-22) we disable requesting classes. For the most part this will
not interfere with your analysis but it can lead to more "unknown"
classes than normal in the heap output.
Java 1.4
Both 1.4.0 and 1.4.1 JVMs seem to have a pretty good implementation of the
JVMPI interface.
Changing the JVM Used with RootCause
Although it is not directly supported by RootCause, it is possible to
change the JVM that will be used at runtime when an application is
run with RootCause. To do this you must
modify the scripts that RootCause builds to run the application.
Unix
On Unix this is the apjava.ksh script. To modify this we must strip off the
first argument and replace it with our java executable. First look for the
exec line, for instance:
exec /opt/rootcause2.1/bin/apjava \
shift
export JAVA_HOME=/opt/jdks/j2re1.4.1_03
exec /opt/rootcause2.1/bin/apjava \
-java=$JAVA_HOME "$@"
When you run, it will use the different JVM you have specified. Note that you
must re-edit the script whenever you re-build the workspace as your edits
will be overwritten.
Windows
At this writing we haven't figured out how to do this in a .cmd file.
Write us at support@ocsystems.com and by then maybe we'll have figured it out.
FAQs for Memory Probes
There are a number of ways. In the runtime parameterization, you can set the tracking start time to some non-zero value. This will delay the tracking of data to allow your application to get through its initialization. It is quite common for a large number of persistent allocations to occur during initialization and since those are single allocations, they do not contribute to application growth. Since the application will run for a much longer time that just the initialization, it is quite safe to ignore the data collection at the start of an application.
Remember to click the build button at the top of the RootCause GUI, even when you just change format parameters. Parameter changes are not committed until you push the build button.
The basic rule is that you should exercise your application with a production type load (or exercise it in production) until it reaches steady state and then run it for a while at steady state. The length of time necessary can vary widely, but an hour should be ok. There is no magic time, the data collected will be correct no matter what length of time that you run it, it is just that the results may not accurately represent what actually happens in the production environment.
One measure of whether you have run long enough is in the summary section. In that section is the time when the last allocation id was encountered and how many allocations had occured up to that point. After this time, no new allocation points were encountered in the run, all the new allocations were from places that had already been encountered. Also in the summary section is the total number of allocations (i.e., the number both before and after the last allocation id time). If the total number of allocations are much greater than the number of allocations when the last allocation id was encountered, then the data is valid.
If this is not true (i.e., the last allocation id was encountered late in the run of the application), then the collected data may or may not be valid. For example, it could be that a new class was loaded very late in the run, or it could be that classes are dynamically loaded (and unloaded) all the time, in which case, new allocation points are discovered all the time.
Another indication of whether your data is "sufficient" is the ratio of the total number of allocations sampled vs the number of unique allocation points. If the total allocations are much larger then the number of unique allocation points, then you can feel that the run was valid.
The memory probes can track memory leaks from JNI code (C++ code called from Java). You can get better tracebacks for the C++ code if you add the shared library to RootCause during the setup of the memory probes for the application as follows.
Use File->Add Dynamic Module to find and add the shared library in the Workspace window. Be sure to do a build afterwards. Now RootCause will do a better job of displaying the tracebacks.