Memwatch Predefined Probe

From OC Systems Wiki!
Jump to: navigation, search


Memory Watch Probe: memwatch.ual

The memwatch.ual predefined probe monitors and gathers data about memory usage to help you detect heap memory leaks.

Heap memory leaks may be identified by tracking the invocations of all known heap manager functions that allocate and deallocate portions of the heap memory, and by matching each such heap memory allocation with its corresponding deallocation.

Snapshots may be taken at any time while the program is running to save the present state of the tracked heap data. This allows the allocation state data that was recorded at various times to be compared to each other and to give an indication of how much heap memory was consumed at those moments. Also, when associated with particular actions of the application program, snapshots can help locate heap usage problems.

The collected data, including any snapshots, are saved in an APD file so that they can be viewed using [AUG_Tools_Reference#|apformat|apformat]] after the program has terminated. Reports are produced showing the time, size, and point of allocation for all items that were still allocated at the time of the snapshot.

NOTE: The memwatch.ual predefined probe is easy to use and is appropriate for determining if your application has a memory leak. For more "industrial-weight" leak tracking look at these other predefined probes: memstat.ual and memleak.ual..


Assumptions

This probe assumes that all requests for allocations and deallocations of dynamic storage are made through calls to discrete run-time heap manager functions (e.g., malloc, free, etc.).

There can be any number of these allocation or deallocation functions, and some functions (e.g., realloc) may do both allocations and deallocations during the same call. Additional heap manager functions may be specified in addition to the default ones by adding your own probes to those provided by memwatch.ual.

Background

Typically, the heap is a large repository of unused memory available for dynamic storage that is controlled by a heap manager. All requests for more memory (especially for dynamically sized objects) needed by a running application program are made to the heap manager. The heap manager carves the heap into smaller portions and allocates those portions upon request. In a well behaved application program, when those allocated portions of heap memory are no longer needed, they are deallocated by returning them to the heap manager for later reuse.

A heap memory leak is an unused and possibly inaccessible portion of heap memory that was allocated but was not subsequently deallocated. If an allocated portion of heap is no longer being used, such as when the only pointer to it goes out of scope or is overwritten, then that portion has probably "leaked." Such heap leaks gradually erode the available heap memory, which may lead to disastrous results when memory runs out.

A true "heap leak" is hard to detect without language and compiler support, because often a program allocates large amounts of data and intentionally keeps it around until program completion. Without tracking every variable to which a heap address is assigned, it is impossible to know when all pointers to a given location are lost. And even if this were possible, there may be other cases where memory should be freed even though it is still potentially accessible.

The memwatch probe helps the user identify heap leaks by recording all allocations and deallocations, and keeping a running total of total allocated memory, and allowing the user to analyze this data to determine if memory usage is appropriate. The reports produced by memwatch are designed to help identify potential leaks by grouping allocations by size, age, and point of allocation.

Usage

This probe is applied at run time using aprobe as described in Ual Parameters below. The configuration file (see Memwatch Configuration File controls the amount and kinds of data collected by this probe.

While most of the probes described in this Appendix provide a point-and-click interface for setting configuration options, the memwatch probe currently does not. The default configuration is generally acceptable, and the heap manager functions to probe are hard-coded into the probe itself.

However, there is a GUI: When the program starts, graphs are displayed which show statistics on memory usage, and allow interactive recording of snapshots showing what data is allocated at that point. This graph is described in detail in Memory Usage Monitor

Memwatch UAL Parameters

memwatch.ual is specified on the aprobe command line or in an APO file as described in Command Line. The specific options are:


aprobe -u memwatch.ual     [-p "[-h] [-v] [-g]  [-c config_filename]"]
    your_program

where:

-c config_filename

specifies that the name of the probe configuration file will follow immediately after -c. The default file name is your_program.memwatch.cfg, where your_program is replaced with the name of your executable program. For example, if your executable program is called wilbur.exe, then the default file name would be wilbur.exe.memwatch.cfg.
-g
starts the Java graphical user interface (GUI) dialog upon probe startup, before running your program. This GUI shows runtime memory usage.
-h
produces brief help text.
-v
means verbose mode, which produces additional progress messages.

Memwatch Configuration File

The example below shows the default memwatch configuration file.


PROBE CONFIGURATION FILE FOR MEMWATCH VERSION 1.0.0

// StartGui can be TRUE (to display the graphical display of
// memory usage during runtime) or FALSE (to turn the display
// off)
StartGui FALSE 
// DepthOfCallChain is a number between 0 and 99 that
// specifies how far back up the call chain we will look to
// distinguish between different allocation points
RuntimeStartTime 0 
// RuntimeStartTime is a number greater than or equal to 0 that
// specifies the number of seconds before tracking will begin.  (Aprobe 4.4.9)
DepthOfCallChain 9 
// IndexCallChains can be TRUE (to display an ID instead of
// a traceback with the mapping between IDs and tracebacks
// being displayed in a separate table)or FALSE (to display
// the full tracebacks each time one is encountered)
IndexCallChains TRUE 
// DisplayReports can take one or both of the following
// options:
//  AgeReport:      Displays a report of each outstanding
//        allocation in a snapshot, starting with the
//        oldest
//  SizeReport:     Displays a list of each allocation point, with
//        the total number of allocations (and size)
//        outstanding for that given allocation point.
//        The list is sorted by size (largest first)
// e.g. DisplayReports AgeReport
//      DisplayReports SizeReport AgeReport
// (Note that you can use DisplayReports NoReport to just get
// summary information)
DisplayReports AgeReport SizeReport 
// Snapshots can be specified using the snapshot keyword with
// the following syntax (multiple entries are
// allowed):
// SNAPSHOT extern:"MyFunction()" On Exit Is "Snapshot name"
// or
// SNAPSHOT extern:"Another()" On Entry Is "Another name"

// Normally all allocations are recorded by the memwatch predefined probe.
// However, there are two kinds of filters that can be applied to control
// which allocations will be included - additive and exclusive.  By
// specifying one or more additive filters, memwatch will record only those
// allocations with matching tracebacks; by specifying one or more exclusive
// filters, memwatch will include every allocation except for those where
// the call chain matches the given exclusion filters.  Mixing the two
// types of filters in the same cfg file is not allowed.
// The filters are easily obtained by cutting and pasting the output
// from a traceback into this cfg file; note that the "==>" parts of
// the traceback are necessary for parsing of the filter. The following
// example shows the correct syntax for an additive filter:
//    Filter extern:"malloc()" in "libc.so"
//       ==&gt extern:"calloc()" + 0x0044 in "libc.so"
//       ==&gt extern:"getcwd()" + 0x0130 in "libc.so"
//       ==&gt extern:"::getCurrentDir(void)" at line 86 (gfile.cc)
//       ==&gt extern:"main()" at line 348 (xpdf.cc)
//       ==&gt extern:"_start()" + 0x00dc
// If you instead wanted an exclusive filter for the same call chain,
// replace "Filter" above with "REMOVE".

Example D-12. memwatch.cfg file

Configuration Variables

RuntimeStartTime

This must be followed by an unsigned integer value that specifies how many seconds to wait before tracking begins. This can be used to avoid tracking objects allocated during program initialization that may be retained through the life of the program. The value must be greater than or equal to 0. The default is 0. (Aprobe 4.4.9)

DepthOfCallChain

This must be followed by an unsigned integer value that specifies how far back up the call chain we will look to distinguish between different allocation points. The value must be within the range of 0 to 99. The default is 9.

IndexCallChains.

This must be followed by the value TRUE or FALSE. The default is TRUE. When this is set to TRUE, each unique traceback is denoted by a unique identification number, and the formatted report tables refer to this ID number rather than the entire traceback. This makes the tables easier to read. A separate list will be reported to show each traceback and its ID number. When this is set to FALSE, the full tracebacks are displayed in the formatted output.

DisplayReports.

This may be followed by any one or more of the following values: NoREPORT, AgeREPORT, or SizeREPORT. The default is both AgeREPORT and SizeREPORT. AgeREPORT causes the formatted output to contain a report showing the outstanding allocations sorted by age, starting with the oldest. SizeREPORT causes the formatted output to contain a report showing the outstanding allocations sorted by size, starting with the biggest. If neither value is specified, then no report is omitted. NoReport is just a placeholder, it does nothing except to specify that you want at least the summary report.

StartGUI

This must be followed by the value TRUE or FALSE. The default is FALSE. The value TRUE indicates that the heap allocation graphs should be shown when the target program runs, even if -g wasn't specified on the command-line. A FALSE value is overridden by the -g command-line option.

Configuration of Filters

FILTER

This must be followed by all the lines of a traceback. Normally, a traceback consists of multiple lines: the first line has the name of a called function, and each subsequent line begins with an arrow and names the immediate caller of the previous line's function. By default, all allocations are recorded by the memwatch predefined probe. But a filter allows you to limit the memwatch probe's recording to only those allocations whose call chain matches all the lines in the filter.

You may specify multiple filters at once. The traceback you need to specify for each filter can be easily obtained by cut-and-paste from a previously formatted output.

Notice that the traceback almost always occupies several lines, and that the "==>" parts of the traceback are necessary at the beginning of each additional line. The following example shows the correct syntax:

FILTER extern:"malloc()" in "libc.so"
==> extern:"calloc()" 0x0044 in "libc.so"
==> extern:"getcwd()" 0x0130 in "libc.so"
==> extern:"::getCurDir(void)" at line 86 (t.cc)
==> extern:"main()" at line 348 (xpdf.cc)
==> extern:"_start()" 0x00dc

If you want an exclusive filter for the same call chain, replace "Filter" above with "REMOVE". You cannot mix additive and subtractive filters.

Configuration of Snapshots

The memwatch probe configuration file allows you to specify the name of some functions for which snapshots are to be automatically taken. This is done with lines beginning with the keyword SNAPSHOT.

Each SNAPSHOT line must specify a particular function as described above. The remainder of the SNAPSHOT line contains pairs, where each pair has a special identifier keyword followed by its own associated value. These pairs give supplementary information about the snapshot.

ON - This optional special identifier must be followed by the value ENTRY or EXIT. These signify that the snapshot is to be taken, respectively, on entry to the function, or upon exit from the function. The default is ON ENTRY.

IS - This optional special identifier must be followed by an arbitrarily long string enclosed within "" quotation marks. It specifies a textual description that is to be logged with the snapshot.

Memory Usage Monitor

When memwatch.ual is invoked with the UAL -g parameter, the Aprobe Memwatch Probe GUI window comes up. By default, this GUI shows two graphs providing information about the amount and rate of heap memory allocations.

In addition to the graphs, there are buttons to control the graphs, and there is a button that allows a snapshot of the current allocation data to be taken interactively.

The graphs display a record of the actual heap activity. The top graph displays the size (number of bytes) of outstanding heap memory allocated over time. Outstanding means that the heap memory is still allocated and has not yet been deallocated. The bottom graph displays the number of allocations that took place during each time interval. The time interval length (in seconds) is shown below.

You can manually zoom in to a selected portion of each graph by dragging the mouse while holding down the CONTROL key on the keyboard. Similarly, you can shift each graph horizontally by dragging the mouse sideways while holding down the SHIFT key on the keyboard. The ResumeUpdates button will restore the graphs back to their normal scale and positions, and resume updating them.

Use the Snapshot button while the target program is running to take snapshots of the current state of recorded heap allocations.

Use the Close button to close the probe and its GUI.

The graphs in this runtime GUI show actual heap memory usage. These graphs are updated periodically, at user-specified intervals. Many allocations and deallocations usually occur within each interval, between updates, so the range of heap sizes is shown for each interval. The HighWaterMark shows the highest value of heap size that was every recorded.

The heap allocation and deallocation events can be examined in more detail, later, by formatting the recorded data to produce reports. The formatted reports will list the data, sorted by age, size, or both. Each snapshot will be reported separately, and each will list all outstanding allocations (allocations which have not be deallocated yet) and all new deallocations since the previous (if any) snapshot was taken.

Memwatch API

You can control the behavior of the memwatch probe by calls from within their own probes. The API for the memwatch probe is defined by [../include/memwatch.h $APROBE/include/memwatch.h]. Some of the functions exported by memwatch.ual are:

ap_Memwatch_Allocation

Record that a heap allocation event has occurred.
ap_Memwatch_Deallocation

Record that a heap deallocation event has occurred.
ap_Memwatch_Reallocation

Record that a heap reallocation event has occurred (which both a deallocation and an allocation event at once).
ap_Memwatch_DoSnapshot

Takes a snapshot of current heap allocations.

Memwatch Demand Actions

(Since 4.4.8).

You can control memwatch.ual using demand.ual and apdemand.

Include demand.ual on the Aprobe command line:

 aprobe -u memwatch -u demand myapp.exe
 

then use apdemand to send actions:

 apdemand snapshot
 

statprof.ual responds to the following actions:

  • memwatch allocations snapshot

Note that all actions containing the action string will be triggered.

Memwatch Performance Issues

The additional execution time caused by the memwatch probe is small (except for snapshot overhead, discussed below). This is because the memwatch probe only instruments a few specific functions, and the probe is tiny compared to the functions themselves (which are usually very computationally intensive).

The memwatch data requires quite a bit of memory. The amount of memory required is proportional to the number of unique tracebacks found among the allocation and deallocation events. This will reduce the memory available to your application program. So, if your application program is close to the process or system memory limit, this could cause its allocations to fail, which could even kill the application.

All the memwatch data that was collected is logged to an APD file when a snapshot is taken. A snapshot is taken either by interactively clicking the GUI button, by calling ap_Memwatch_DoSnapShot(), or by default at program exit. You can see that each snapshot may consume many megabytes, and it can take some time for the probe to write all this data to APD files on disk. The probe shares the same process as your application program, so we suggest that you take a snapshot only if the delay caused by writing out this data will not change your program's behavior.