Memstat Predefined Probe

From OC Systems Wiki!
Jump to: navigation, search


Statistical Memory Analysis Probe

The memstat.ual predefined probe uses statistical sampling to monitor memory usage to help you detect heap memory leaks.

Heap memory leaks may be identified by tracking the invocations of all known heap manager functions that allocate and deallocate portions of the heap memory, and by matching each such heap memory allocation with its corresponding deallocation. Rather than tracking all allocations and deallocations, which can cause performance problems, memstat.ual will sample allocations and deallocations over a longer period of time.

The collected data are saved in an APD file so that they can be viewed using apformat after the program has terminated. Reports are produced showing the time, size, and point of allocation for all items that were still allocated at the time of the snapshot.

NOTE: The memstat.ual predefined probe is most appropriate for very long running programs or programs where you need a very light touch. The statistical approach used by memstat.ual accomplishes this. For shorter running programs, you may want to use memleak.ual.

Assumptions

This probe assumes that all requests for allocations and deallocations of dynamic storage are made through calls to discrete run-time heap manager functions (e.g., malloc, free, etc.).

There can be any number of these allocation or deallocation functions, and some functions (e.g., realloc) may do both allocations and deallocations during the same call.

Background

Typically, the heap is a large repository of unused memory available for dynamic storage that is controlled by a heap manager. All requests for more memory (especially for dynamically sized objects) needed by a running application program are made to the heap manager. The heap manager carves the heap into smaller portions and allocates those portions upon request. In a well behaved application program, when those allocated portions of heap memory are no longer needed, they are deallocated by returning them to the heap manager for later reuse.

A heap memory leak is an unused and possibly inaccessible portion of heap memory that was allocated but was not subsequently deallocated. If an allocated portion of heap is no longer being used, such as when the only pointer to it goes out of scope or is overwritten, then that portion has probably "leaked." Such heap leaks gradually erode the available heap memory, which may lead to disastrous results when memory runs out.

A true "heap leak" is hard to detect without language and compiler support, because often a program allocates large amounts of data and intentionally keeps it around until program completion. Without tracking every variable to which a heap address is assigned, it is impossible to know when all pointers to a given location are lost. And even if this were possible, there may be other cases where memory should be freed even though it is still potentially accessible.

The memstat probe helps the user identify heap leaks by recording all allocations and deallocations, and keeping a running total of total allocated memory, and allowing the user to analyze this data to determine if memory usage is appropriate. The reports produced by memstat are designed to help identify potential leaks by grouping allocations by size, age, and point of allocation.

Usage

This probe is applied at run time using aprobe as described in Memstat UAL Parameters below. The configuration file (see Memstat Configuration File controls the amount and kinds of data collected by this probe.

While most of the predefined probes provide a point-and-click interface for setting configuration options, the memstat probe currently does not. The default configuration is generally acceptable, and the heap manager functions to probe are hard-coded into the probe itself.

Memstat UAL Parameters

memstat.ual is specified on the aprobe] command line or in an APO file as described in Command Line. The specific options are:

aprobe -u memstat.ual    [-p [-h] [-v] [-a] [-b] [-c config_filename] [-d num] [-e secs] [-f sub-option] [-g] [-i num] [-k num] [-m num] [-p] [-r secs] [-s secs] [-t secs] ]  your_program

where:

-a
report each allocation and free.
-b
don't track/report deallocation points.
-c config_filename
specifies that the name of the probe configuration file will follow immediately after -c. The default file name is your_program.memstat.cfg, where your_program is replaced with the name of your executable program. For example, if your executable program is called "wilbur.exe", then the default file name would be "wilbur.exe.memwatch.cfg".
-d num
set traceback depth to num.
-e secs
End virtual time for tracking new allocations (format-time).
-f sub-options
where sub-options are:
-d "name" - directory for default output files.
-e "name" - details output filename.
-o "name" - object dump filename.
-s "name" - summary output filename.
-t "name" - traceback output filename.
-h
produces brief help text.
-v
means verbose mode, which produces additional progress messages.

Memstat Configuration File

The example below shows the default memstat configuration file.

PROBE CONFIGURATION FILE FOR MEMSTAT VERSION 1.0.0


// 1. Runtime options:
//
// 1.1. DepthOfCallChain <num>
//    Controls the maximum number of entries in each call chain recorded
//    by this probe.
//
// 1.1. PeriodicStatsInterval <num> (default 1)
//    Sets the number of seconds to be used as an interval for data summary.  
//
//    For example:
//
//    PeriodicStatsInterval 1
//
//    Will report allocation statistics every second.  
// 
//    PeriodicStatsInterval of 0 turns off reporting periodic statistics.
//
// 1.2. Verbose TRUE | FALSE
//
//    Will turn on reporting of periodic intervals expiration.
//
// 1.3. FreeCacheSize <num> (default 0)
//
//    Defines a number of objects deallocated with a call to free()
//    that this probe will hold on to before giving them back to the heap.
//    Larger FreeCacheSize  values make it detection of double deallocations
//    more likely, increasing the memory consumption by the application
//    at the same time.
//
// 1.4. Options to control sampling and size filtering
//
// 1.4.1. SamplingRatio <num> (default 100)
//
//    Defines the ratio of the number of allocations to the number of samples.
//    For instance, specifying 1000 means that only one in a 1000 allocations
//    are actually sampled (but see the filtering criteria below). Specifying
//    a ratio of 1 means that every allocation is recorded.
//
// 1.4.2. MaxItemsToTrack <num> (default 100000)
//
//    When sampling is on (e.g. the ratio is > 1), this contains the maximum
//    number of objects we will track. If an object is not entered into the
//    table, a corresponding deallocation or reallocation will not be logged.
//    If the table size is reached, you will get a single error message
//    informing you of this; further allocations will be recorded once there
//    is free space. Note that setting this to 0 has the same behavior as
//    disabling sampling.
//
// 1.4.3. FilterOutBelowSize <num> (default is 0)
//
//    When sampling is on, ignore the allocation if it is smaller than the
//    specified size.
//
// 1.4.3. FilterInAboveSize <num> (default is UINT_MAX == 4294967295U)
//
//    When sampling is on, add this allocation to the current list if it is
//    greater than the specified size regardless of how many objects are in
//    the table. Use this option carefully since it could lead to very large
//    active object tables if you set it too low.
//
// 1.5. AllocationModule modulename
//    For Solaris and Linux you can override the use of libc.so as the 
//    module for locating the malloc and free routines. For instance, if you
//    use the libmapmalloc.so library, you would use:
//
// AllocationModule "libmapmalloc.so"
//
//    This option is ignored (with a warning) for other platforms.
//
// 1.6. TrackingEnabledInitially TRUE | FALSE
//
//    Will turn on or off tracking at program start-up.  Default is TRUE.
//
//
//
// 2. Formattime options:
// 2.1. Reports To Produce (DisplayReports)
//    DisplayReports \
//       ReportAllAllocations \
//       ReportPeriodicStats \
//       ReportDoubleDeallocations \
//       ReportAllocationPoints \
//       ReportDeallocationPoints \
//       PlotPeriodicStats \
//       ShowTimeSliceSummary \
//       ReportAnalysis
//
// 2.1.1. ReportAllAllocations
//    At runtime memstat probe collects information about ALL allocations 
//    made by your application.  This includes the address of 
//    allocated/deallocated memory its size, location (traceback) of 
//    the allocation/deallocation point.  By default this data will be
//    summarized for you in several reports that groups the data
//    by allocation/deallocation points.  Sometimes you may want to see
//    the raw allocation/deallocation data as it was recorded at runtime.
//    Selecting ReportAllAllocations report and reformatting the data
//    will accomplish that.  Be aware of large amounts of data this report
//    may produce.
//
// 2.1.2. Report Periodic Statistics (ReportPeriodicStats)
//    Choosing this report will print the memory allocation statistics
//    collected at periodic intervals.  This report will not have any affect
//    unless PeriodicStatsInterval runtime option (-s <num> on the command line)
//    was used at runtime.  This report may be useful in seeing how
//    the allocation activity relates to the time line.
//
// 2.1.3. PlotPeriodicStats
//    Choosing this report will result in a plot of all periodic statistics
//    collected at runtime (see 2.1.2 ReportPeriodicStats).
//
// 2.1.4. ReportDoubleDeallocations
//    This option is on by default.  When the option is on you will be
//    notified of all deallocations for which an allocation has not been
//    seen yet.  In some rare circumstances it is possible to have legitimate
//    deallocations of memory for which aprobe has not detected an allocation.
//    In all of these cases the deallocation is done by some system
//    libraries and usually could be ignored.  In particular their is initial
//    allocation done by a Java VM which happens when Aprobe has probes
//    disabled and they can be safely ignored. The double deallocations
//    coming from the user code are almost always legitimate problems, which
//    should be addressed. 
//    There are actually two double deallocation cases that are detected by
//    this probe: one is when a memory is deallocated, for which have not
//    seen a single allocation, and two is the case when we see memory
//    being deallocated after having been deallocated already.
//    The error message for the first one is:
//       "freeing memory that has not been allocated"
//    The error message for the second one is:
//       "freeing memory that has been deallocated"
// 
//    ReportDoubleDeallocations only controls reporting of the first type
//    of double deallocation error, since the second one should never be
//    ignored.
//
// 2.1.5. ReportAllocationPoints
//    This option is on by default.  It is the central report produced
//    by this probe.  This report produces a summary of allocations 
//    grouped by allocation points.
//
// 2.1.6. ReportDeallocationPoints
//    This option is on by default.  This report produces the summary
//    of all deallocation points known for each allocation point.
//    It is very useful when you try to understand the data flow between
//    the allocation and deallocation points.
//
// 2.1.7. ShowTimeSliceSummary
//    This option summarizes the outstanding allocations over the lifetime
//    of the application. Each allocation is placed into a slice based on the
//    allocation time. The allocation time slice is specified with the
//    TimeSliceDuration option. The MaxTimeSlices controls the maximum number
//    of these slices that are displayed. Note that the TimeSliceDuration may
//    be modified dynamically to allow more samples to fit into the array.
//
// 2.1.8. ReportAnalysis
//    This option presents the results of the analysis performed on the data
//    to attempt to identify which allocation points you should look at first.
//    This option is on by default.
//
// 2.2 Allocation Points
//    This probe records Allocation Points with tracebacks to describe where
//    a given memory allocation was made.  Labels to help identify these
//    tracebacks, as well as special filters to only include those tracebacks
//    which you are interested in monitoring, are available.
//
// 2.2.1 AllocPoint (Allocation Points)
//    You can give meaningful names to symbolic tracebacks produced by this 
//    probe with the AllocPoint directive.  The AllocPoint keyword has to be
//    followed on the same line by the name/comment you would like to associate 
//    with a given traceback, and the traceback itself has to be provided
//    on the lines following the AllocPoint directive.  The tracebacks have
//    to start with keywords: "==>" or "Filter".  For example:
//
//   AllocPoint "Known leak #200"
//   Filter extern:"malloc()" in "MSVCRT.dll"
//   ==> extern:"spaghetti()"
//
// 2.2.2 Filtering Allocation Points
//    Allocation Points can also be excluded from recording, by providing
//    a matching traceback to filter; the syntax for specifying exclusive
//    filters is the same as that for additive filters, but with the keyword
//    "REMOVE" in place of "Filter":
//
//   REMOVE extern:"malloc()" in "MSVCRT.dll"
//   ==> extern:"spaghetti()"
//
// 2.3. StartTime <Seconds> /StopTime <Seconds>
// 
//    These options allow for selective formatting of data collected
//    between the two intervals specified with StartTime and StopTime options.
//
//    Example:
//    StartTime 10
//    StopTime 20
//
//    The above options will cause your report to only include data
//    during seconds 10 - 20 of your program run.
//    You can specify these options on the command line with
//    arguments -t <num> and -e <num> to memstat.
//
// 2.4. ShowFreedAllocationsInSummary
//    By default the summary table(s) do not show allocation points where
//    all of the allocations have been freed. Setting this option to TRUE
//    will include such freed allocations in the summary.
//
// 2.5. MaxTimeSlices
//    This is used with the ShowTimeSliceSummary option to control the maximum
//    slices we hold. Be careful about making this number too high because it
//    will directly influence the amount of memory required to format the
//    data.
//
// 2.6. TimeSliceDuration
//    This is used with the ShowTimeSliceSummary option to control the size of
//    each slice. Using a smaller number allows a finer granularity but may
//    move more allocations into the last time slice if MaxTimeSlices is
//    reached.
//
// 2.7. DisplayDeltaTimes
//    This option is on by default. It displays each allocation, free, etc.
//    as a delta in seconds from the beginning of the program. If this is
//    turned off (DisplayDeltaTimes FALSE) it will display the actual times.
//
// 2.8. OutputFile
//    This keyword sets the output file(s) that the reports are directed to.
//    By default all of the reports go to standard out. The option can set
//    a directory that all files go to or can set individual files. Any files
//    that are not specified will go to stdout. Note that the directory must
//    already be created and must be writable. The Directory option cannot
//    appear with any of the individual file options - the first option found
//    will take precedence.
//
//    The sub-options are:
//
//    Directory "Name"     - specify an output directory. Three files will be
//                           created of the form xxx.memstat.yyy.txt where 
//                           xxx is the application name and yyy is "summary",
//                           "details" and "traceback" for the three files.
//
//    SummaryFile "Name"   - specify the summary output filename. All of the
//                           summarized tables and header information goes into
//                           this file.
//
//    DetailsFile "Name"   - specify the details output filename. All of the
//                           detailed allocation data and periodic data goes
//                           into this file.
//
//    TracebackFile "Name" - specify the traceback output filename. All of the
//                           tracebacks that were found go into this file.
//
// 2.9. PlotAllocation ID
//    This will plot an individual graph for the given traceback ID showing
//    the outstanding allocation size over time. You may have multiple instances
//    of this keyword. Note that traceback IDs will only remain the same
//    between one apformat and the next if the same set of APD files is used
//    and the start and stop times are used.
//
// 2.10. ShowZeroGrowthIds
//     By default, the analysis summary does not display entries for
//     allocation points that exhibited zero growth. Turning this option on
//     will display those entries.
//
// 2.11. AnalysisPercentage
//     The percentage of the formatted data used for analysis. Setting this
//     to a higher value than the default (80) will look at earlier data in
//     the run with an increased risk of marking initial allocations as growth.
//
// 2.12. DisplayAnalysisWithTracebacks
//     This option displays any analysis results with the traceback as well
//     as in the table. Note that if you have requested a detailed report you
//     will see tracebacks for analyzed allocations twice - once with the
//     analysis data and once in the full list of tracebacks. This option is
//     on by default.
//
// 2.13. RuntimeStartTime
//     By default tracking of object creation occurs from the start of the
//     application. By setting this option to a value (in seconds) you can
//     delay the tracking, speeding up initialization. For many applications,
//     the allocations made during initialization are not representative of the
//     steady-state behavior and are already filtered out by the analysis
//     algorithms. There is no optimal value for this - it depends entirely
//     on your application.
//
// 2.14. MaxDisplayTimeSlices
//     This sets the number of time slices you see presented if the
//     interval summary report is enabled. If this is less than the number of
//     slices recorded, you will see the last 'n' slices.
//
// 2.15. TimeSliceFields
//     This sets the fields displayed in the interval summary report.
//     It is followed by one or more of the following sub-options. You can
//     specify this option multiple times if desired.
//
//        AllocSizeField      : Size of allocations in this interval
//        FreedSizeField      : Size of frees in this interval
//        DeltaSizeField      : Size difference in this interval
//        TotalSizeField      : Total outstanding size
//        TotalDeltaSizeField : Delta outstanding size from last interval
//        NumAllocsField      : Number of allocations in this interval
//        NumFreesField       : Number of frees in this interval
//        DeltaNumField       : Number difference in this interval
//        TotalNumField       : Total outstanding number
//        TotalDeltaNumField  : Delta outstanding number from last interval
//
// 2.16 LimitAllocPoints
//     This limits the number of allocation points included in the reports.
//     It is followed by an integer value which specifies the maximum 
//     number of allocation points to summarize.  The detailed report is
//     not affected by this setting.
//
// 2.17 LimitAllocPercentage
//     This limits the number of allocation points included in the reports
//     to the number of allocation points displaying the user defined
//     percentage of the total number of bytes leaked.  It is followed
//     by an integer value indicating the limiting percentage of the total
//     bytes leaked.  This setting does not affect the growth analysis 
//     report, or the detailed report.
//
//     Ex: LimitAllocPercentage 85 - limits the report to those allocation
//         points contributing to 85 percent of the total bytes leaked.

Example. memstat.cfg file

Configuration Variables

PeriodicStatInterval

This must be followed by an unsigned integer value that specifies the interval in seconds between runtime statistics reporting. The value 0 will disable reporting. 0 is the default.

Verbose

This must be followed by TRUE or FALSE. The value TRUE enables reporting of sampling interval expirations. The default is FALSE.

FreeCacheSize

This must be followed by an unsigned integer. This value defines a number of objects deallocated with a call to free() that this probe will hold on to before giving them back to the heap. Larger FreeCacheSize values make it detection of double deallocations more likely, increasing the memory consumption by the application at the same time. The default value is 0.

SamplnigRatio

This must be followed by an unsigned integer create than 0. This value defines the ratio of the number of allocations to the number of samples. For instance, specifying 1000 means that only one in a 1000 allocations are actually sampled (but see the filtering criteria below). Specifying a ratio of 1 means that every allocation is recorded. The default is 100.

MaxItemsToTrack

This must be followed by an unsigned integer. When sampling is on (e.g. the ratio is > 1), this value specifies the maximum number of objects we will track. If an object is not entered into the table, a corresponding deallocation or reallocation will not be logged. If the table size is reached, you will get a single error message informing you of this; further allocations will be recorded once there is free space. Note that setting this to 0 has the same behavior as disabling sampling. the default value is 100000.

FilterOutBelowSize

This must be followed by an unsigned integer. When sampling is on, allocations smaller than this value in size will be ignored. The default value is 0 (no filtering).

FilterInAboveSize

This must be followed by an unsigned integer. When sampling is on, an allocation will be added to the current list if it is greater than the specified size regardless of how many objects are in the table. Use this option carefully since it could lead to very large active object tables if you set it too low. The default value is 4294967295.

AllocationModule

This must be followed by a module name in quotes. For Solaris and Linux you can override the use of libc.so as the module for locating the malloc and free routines. For instance, if you use the libmapmalloc.so library, you would use:

 AllocationModule "libmapmalloc.so"
 

This option is ignored (with a warning) for other platforms. The default value is "libc.so".

TrackingEnabledInitially

This must be followed by TRUE or FALSE. When FALSE tracking will be turned off at program start-up. The default is TRUE.

DisplayReports

This may be followed by one or more of the following report names: ReportAllAllocations, ReportPeriodicStats, ReportDoubleDeallocations, ReportAllocationPoints, ReportDeallocationPoints, PlotPeriodicStats, ShowTimeSliceSummary, and ReportAnalysis.

 The default is ReportDoubleDeallocations ReportAllocationPoints ReportAnalysis.   See the example configuration file for details of each report.

AllocPoint

This must be followed on the same line by the name/comment you would like to associate with a given traceback, and the traceback itself has to be provided on the lines following the AllocPoint directive. The tracebacks have to start with keywords: "==>" or "Filter". For example:

   AllocPoint "Known leak #200"
   Filter extern:"malloc()" in "MSVCRT.dll"
   ==> extern:"spaghetti()"
 

Allocation Points can also be excluded from recording, by providing a matching traceback to filter; the syntax for specifying exclusive filters is the same as that for additive filters, but with the keyword "REMOVE" in place of "Filter":

   REMOVE extern:"malloc()" in "MSVCRT.dll"
   ==> extern:"spaghetti()"
 
StartTime

This keyword must be followed by an unsigned integer. This specifies the number of seconds from program start when reporting should begin. The default value is 0. You can specify this option on the command line with arguments -t num. This value is used at format-time during analysis and reporting, not at runtime.

StopTime

This keyword must be followed by an unsigned integer. This specifies the number of seconds from program start when reporting should stop. A value of 0 indicates no stop time. The default value is 0. You can specify this option on the command line with arguments -e num. This value is used at format-time during analysis and reporting, not at runtime.

ShowFreedAllocationsInSummary

This must be followed by TRUE or FALSE. By default the summary table(s) do not show allocation points where all of the allocations have been freed. Setting this option to TRUE will include such freed allocations in the summary.

MaxTimeSlices

This must be followed by an unsigned integer greater than 0. This is used with the ShowTimeSliceSummary option to control the maximum slices we hold. Be careful about making this number too high because it will directly influence the amount of memory required to format the data.

TimeSliceDuration

This must be followed by an unsigned integer greater than 0. This is used with the ShowTimeSliceSummary option to control the size of each slice. Using a smaller number allows a finer granularity but may move more allocations into the last time slice if MaxTimeSlices is reached.

DisplayDeltaTimes

This must be followed by TRUE or FALSE. This option is on by default. It displays each allocation, free, etc. as a delta in seconds from the beginning of the program. If this is turned off (DisplayDeltaTimes FALSE) it will display the actual times.

OutputFile

This may be followed the sub-options: Directory, SummaryFile, DetailsFile, and TracebackFile. This keyword sets the output file(s) that the reports are directed to. By default all of the reports go to standard out. The option can set a directory that all files go to or can set individual files. Any files that are not specified will go to stdout. Note that the directory must already be created and must be writable. See the example configuration file for more information.

PlotAllocation ID

This must be followed by a plot ID. This will plot an individual graph for the given traceback ID showing the outstanding allocation size over time. You may have multiple instances of this keyword. Note that traceback IDs will only remain the same between one apformat and the next if the same set of APD files is used and the start and stop times are used.

ShowZeroGrowthIds

This must be followed by TRUE or FALSE. By default, the analysis summary does not display entries for allocation points that exhibited zero growth. Turning this option on will display those entries.

AnalysisPercentage

This must be followed by an integer in the range 1 to 100. This is the percentage of the formatted data used for analysis. Setting this to a higher value than the default (80) will look at earlier data in the run with an increased risk of marking initial allocations as growth.

DisplayAnalysisWithTracebacks

This must be followed by TRUE or FALSE. This option displays any analysis results with the traceback as well as in the table. Note that if you have requested a detailed report you will see tracebacks for analyzed allocations twice - once with the analysis data and once in the full list of tracebacks. The default is TRUE.

RuntimeStartTime

This must be followed by an unsigned integer. By setting this option to a value (in seconds) you can delay the tracking, speeding up initialization. For many applications, the allocations made during initialization are not representative of the steady-state behavior and are already filtered out by the analysis algorithms. There is no optimal value for this - it depends entirely on your application. This value affects tracking at runtime.

MaxDisplayTimeSlices

This must be followed by an unsigned integer. This sets the number of time slices you see presented if the interval summary report is enabled. If this is less than the number of slices recorded, you will see the last 'n' slices.

TimeSliceFields

This must be followed by one or more of the following options: AllocSizeField, FreedSizeField, DeltaSizeField, TotalSizeField, TotalDeltaSizeField, NumAllocsField, NumFreesField,DeltaNumField, TotalNumField, or TotalDeltaNumField. This sets the fields displayed in the interval summary report. See the example configuration file for more information.

LimitAllocPoints

This must be followed by an unsigned integer. This limits the number of allocation points included in the reports. The detailed report is not affected by this setting.

LimitAllocPercentage

This must be followed by an unsigned integer value between 0 to 100. This limits the number of allocation points included in the reports to the number of allocation points displaying the user defined percentage of the total number of bytes leaked. This setting does not affect the growth analysis report, or the detailed report. For example:

 LimitAllocPercentage 85
 

limits the report to those allocation points contributing to 85 percent of the total bytes leaked.

Memstat API

You can control the behavior of the memstat probe by calls from within their own probes. The API for the memstat probe is defined by $APROBE/include/memstat.h. The functions exported by memstat.ual are:

ap_Memstat_Enable

Enable memory tracking.
ap_Memstat_Disable

Disable memory tracking.

Memstat Demand Actions

(Since 4.4.8).

You can control memstat.ual using demand.ual and apdemand.

Include demand.ual on the Aprobe command line:

 aprobe -u memstat -u demand myapp.exe
 

then use apdemand to send actions:

 apdemand enable memstat
 

statprof.ual responds to the following actions:

  • enable memstat data collection
  • disable memstat data collection

Note that all actions containing the action string will be triggered.

Memstat Performance Issues

The additional execution time caused by the memstat probe is small (except for snapshot overhead, discussed below). This is because the memstat probe only tracks a statistical sample of allocations rather than all allocations, which requires a longer run time. This sampling ratio can be adjusted to reduce overhead (and APD space) further at the cost of requiring longer run times, or decreased to require shorter run times at the cost of more overhead (and APD space).

The sampling ratio also affects the size of allocation tables needed to track allocations. This will reduce the memory available to your application program. So, if your application program is close to the process or system memory limit, this could cause its allocations to fail, which could even kill the application.

All the memstat data that was collected is logged to an APD file at program exit.