AUG Predefined Probes

Next Previous Top Contents [Index

Aprobe User Guide

Predefined Probes and Libraries Reference

---

Overview

The Aprobe product includes ready-to-use UALs built from probes written by OC Systems. We call these "predefined probes". These may be referenced by name on the aprobe command line and applied to any application, without writing any APC or using the apc command.

This appendix provides more in-depth information about how the major predefined probes work and how to use them.

coverage.ual

records the execution of each known source line in the specified functions of the executable program and produces a report indicating which lines in each function were not executed. See coverage.ual.

events.ual

records the start and end times of program functions and user-defined "events", organizes them by call tree, and provides several reports which help to analyze the performance of an application. This probe supports Java as well as native programs. See events.ual.

profile.ual

monitors the selected function calls in the executable program and displays execution time statistics about those functions to help indicate where and when the program is spending its wallclock time. See profile.ual.

statprof.ual

uses the Unix kernel sampling mechanism for performance analysis (as used by prof) to provide statistical information about where the application is spending its CPU time. See statprof.ual

trace.ual

traces the execution of the selected functions within an application program to record when, and in which order, and by which thread each was invoked. This probe supports Java as well as native programs. See trace.ual. The trace.ual predefined probe is used in the on-line example $APROBE/examples/evaluate/4.predefined. Note that direct use of this probe, especially using the GUI to define what to trace, has been superceded by the RootCause interface.

memstat.ual

reports heap memory allocation patterns and leaks using statistical sampling. See memstat.ual.

memwatch.ual

reports heap memory allocation patterns and leaks. See memwatch.ual.

info.ual

prints information about the executable program and/or an APD file. It can be used with either [aprobe-9.html#MARKER-9-1075 aprobe] or [aprobe-9.html#MARKER-9-1035 apformat] to display the symbol names within the executable and its shared libraries, or the UALs and threads recorded in an APD file. This probe is used to implement the [aprobe-9.html#MARKER-9-1052 apinfo] and [aprobe-9.html#MARKER-9-1110 apsymbols] commands. See [#MARKER-9-2205 "Info Probe: info.ual"].

Lastly there is

quick_gui.ual

a library of functions to provide simple Java GUI support to a user-written probe. The interface is defined in $APROBE/include/quick_gui.h, and includes histogram and plot graph support, as well pop-up Yes/No and message dialogs. Use of this library is illustrated by the $APROBE/examples/learn/visualize_data example. See [#MARKER-9-2220 "quick_gui Library"].

In most of this appendix, we'll refer to these by their simple filename: coverage, events, profile, trace, memwatch, etc.

How They Work

The predefined probes are written using the features and interfaces described in this user's guide, and could have been written by a user. As such, they demonstrate the power and flexibility of Aprobe. But make no mistake--they are complex, and are not recommended as models or examples for learning Aprobe. Nonetheless, it is useful to understand what happens when you apply one of these probes, particularly as it relates to performance of your application and your choice of options.

When aprobe is invoked on an application, it loads the executable, the libraries it references, and the specified UALs and reads in all the symbol table information. The aprobe runtime applies all the statically defined probes described in the UALs, then invokes any probe program on_entry actions. It is in such a probe that all the start-up actions for a predefined probe occur: UAL command-line arguments are read; the configuration file is read and recorded; and, for those probes that support it, the Java GUI process is started if requested. Using the GUI, the user may request a list of all the symbols in a module from which to select those to be probed, and specify other options.

When the final configuration is set and execution is continued, the probes on the functions specified in the configuration file are applied. When all probes have been applied successfully, and any other probe program and probe thread on_entry actions have been executed, the application program continues its normal execution.

As the application executes, the probes are invoked and collect or update the data according to their function. If snapshots are used, the data is written to the APD file at that point -- otherwise the data is written out when execution finishes.

When execution is completed, the APD files created by the probe contain the data collected. Using apformat on these produces the data collected which of course varies with each probe.

Performance Issues

As the application executes, there may be thousands of different probes, executed millions of times, all collecting data. The memwatch probe will only probe a few system functions that manipulate heap memory, but these may be invoked thousands of times. The coverage, profile, and trace probes are invoked for every function (and perhaps every line of every function) that was specified in the probe's configuration file. If data is logged directly to an APD file on disk (as the trace probe can do), this may be millions of bytes of I/O being written. If data is collected in memory, as is done by the memwatch probe, a long-running or large program may require a lot of virtual memory.

The areas where performance of a probe is noticed are:

1. Aprobe (and apformat) start-up, when the symbol tables are read from the target application and its libraries;
2. The Configuration GUI, when lists of symbols must be built for display and manipulated by the interface;
3. Instrumentation, when probes are patched into the entry, exit, and perhaps each line of the user-selected list of functions.
4. Execution, when the probes are invoked and executed as an extension to the user's program;
5. Data collection, when the probes record their data in memory, and perhaps write it to disk; and
6. Data formatting, when the data that has been logged is analyzed.

The perceived speed of items (1) and (2) is directly related to the size of the application, or more specifically, the number of symbols in the application. The instrumentation time (3) is directly proportional to the number of probes being applied, which for the coverage, profile, and trace probes means the number of functions selected in the configuration file or GUI. The instrumentation time for the memwatch and statprof probes is constant and fast. The execution overhead (4) of the probes once they're started depends on the number of probes actually invoked, which is a product of the number of functions selected by the user and the number of times each such function is called. The data collection and formatting times (5, 6) vary greatly for each probe, and are described in the Performance Issues section for each probe. In brief, however:

• The memwatch probe records data for each memory allocation, proportional to the amount of memory allocated by the application under test.
• The coverage probe simply increments an integer for each line executed, so the amount of data is a constant multiple of the number of lines being covered.
• The profile probe simply adds to a total execution time kept for each function, so the data is proportional to the number of functions being profiled.
• The trace probe may be very data intensive at run-time unless load-shedding is enabled, or the fixed-sized circular buffer log method is specified, since every entry and exit (and optionally every line) records many bytes of data.
• The events probe is similar to the trace probe in overhead. In fact trace may be used in conjunction with events in order to provide load-shedding, even if nothing was traced.

Common Interfaces

The behavior of a predefined probe may be controlled in a number of ways:

• by passing parameters to the UAL on the aprobe command line;
• by editing a text "configuration file"
• by interacting with a Java Graphical User Interface (GUI) controlled by the probe;
• by providing your own probe which explicitly calls functions exported from the predefined probe.

These mechanisms are much the same for the dynamic analysis probes above.The info.ual predefined probe is a utility for which all behavior is specified on the command-line; and quick_gui.ual is a library, so all interaction with it is by function calls from within a user-written probe. The statprof.ual predefined is another simple probe that does not require a configuration file or GUI to customize its behavior.

Command Line

In the on-line "4.predefined" example, a command line like the following is used:

aprobe -u trace.ual -p "-g -v" main

The trace.ual file is actually located in the directory $APROBE/ual_lib, but this directory searched by default for any UAL not found in the current working directory, so you only need to provide the base name of the UAL file.

The -p option which follows trace.ual indicates that the next argument is a string of options to be passed to the UAL. In this case, those options are -g, which indicates that the Java Graphical User Interface (GUI) is to be started; and -v, which causes "verbose" informative output to be generated by the probe. Every predefined probe accepts the option -h, which describes the full set of command-line options available to that probe. So, to display the full set of options for trace.ual, execute:

aprobe -u trace.ual -p -h main

Predefined probes are generally initiated by invoking [aprobe-9.html#MARKER-9-1075 aprobe] explicitly. However, they may also be invoked implicitly by linking your application with libdal.so or using the run_with_aprobe_apo script and specifying the command-line options in a .apo file as described in [aprobe-7.html#MARKER-9-868 Chapter 4, "Loading Probes Without aprobe"].

Configuration File

The configuration file is used to designate which functions in the target application are to be analyzed, and to specify other options. The easiest way to get an initial configuration file is simply to invoke the predefined probe with no options. You'll get a warning about not finding a configuration file, but one will be created.

You can use the [#MARKER-9-1961 Configuration GUI] to create or change the configuration file by specifying the -g option. This provides a point-and-click interface in which to choose your functions and other options and save them in a configuration file. (Note that this GUI can be quite slow for very large applications, so editing the file directly may be more practical in such cases.)

The default name for the probe configuration file is your_program.probe_name.cfg, where your_program stands for the name of your executable application program, and probe_name is the basename of the predefined probe, e.g., trace. For example, if your program is called wilbur.exe and you're using trace.ual, then the default configuration file name would be wilbur.exe.trace.cfg.

The format of the a probe configuration file is:

PROBE CONFIGURATION FILE FOR probe_name VERSION 2.0.0
keyword data
...
keyword data
// This is a comment line

A warning will be given if the very first line of this file isn't valid, but an attempt will be made to use the information provided. The only allowed variations on this header line are the number of blank spaces between the words, and the case of each character.

All keywords in the configuration file are case-insensitive. However, all function or module names are always case-sensitive.

Each line begins with a keyword, or is a continuation line. A line is a continuation line if the previous line ends in a backslash ("\"). The continuation character is treated as a word separator, like a blank.

Comment lines can be interspersed with any other lines. Comment lines are those lines whose first 2 non-blank characters are "//". Comment lines are ignored by probe and are intended only for the benefit of the human reader.

Any invalid line is ignored. An invalid line is one which contains an invalid keyword or value, or has a missing or extraneous word.

Configuration of Selected Functions

The coverage, profile, and trace predefined probes operate by probing functions in the user's application, so the names of those functions must be specified in the configuration file. While each predefined probe uses different keywords to designate which functions to probe, the syntax of the function name itself is the same in all cases. It is exactly the same as the syntax used for a probe itself, as described in [aprobe-6.html#MARKER-9-497 "Specifying Function Names"] and repeated here:

[ [file_name | extern] : ] function_name [ in module_name ]

file_name, function_name, and module_name are all strings that are bounded by quotation marks (that is, ""), which allows each name to contain embedded blank space characters. The file_name and the module_name are optional. If module_name is omitted, the application module is assumed, and the special character "*" can be used to designated every module. The file_name is required only for static (file-local) functions. A colon must exist between the file_name and the function_name. The colon may be surrounded by optional whitespace, and the keyword IN must always be surrounded by whitespace. Examples are:

"my_file":"my_function"
extern:"malloc()" in "libc.so"

In the configuration file for trace.ual, the keyword indicating that a function is to be traced is "Trace", where the cases of the letters is not significant. In the main.trace.cfg file, then should appear one or more lines that look like

Trace "my_file":"my_function"

Predefined probes may have other keywords, such as Trigger, Remove and Snapshot, which are followed by function names. These are described in their corresponding section below. And, note that memwatch.ual does not require specifying application functions this way -- see [#MARKER-9-2173 "Memory Watch Probe: memwatch.ual"].

Configuration GUI

Some, but not all, predefined probes offer a Configuration GUI (Graphical User Interface). The coverage, profile, and trace predefined probes do offer it, but not events, memwatch, nor info

If the -g option is specified to the UAL, the probe starts a Java process to run the Configuration GUI. This provides a point-and-click interface for selecting functions and setting other options which are recorded in the [#MARKER-9-1955 Configuration File]. This interface differs for each probe, but generally consists of several panes in which functions and option settings are selected. At the bottom are Save & Run , Run, Save As, Abort, and Help buttons. Run proceeds with the execution of the program under aprobe with the current configuration; Save updates the configuration file on disk with the current settings; Abort quits the GUI and aborts the aprobe and target application processes; and Help brings up a dialog describing each pane and the other buttons.

Function Selection Interface

The Configuration GUI can be useful for selecting the functions that are to be operated upon by the probe. The "4.predefined" example illustrates this for the trace probe, and this process generally applies to the other probes as well:

Click on the "Pick..." button under the top pane (for Trace, the top pane is labeled as "Traced Functions"). You'll be presented with a list of modules (the executable plus any shared libraries it loads). When you double-click on one of those, it will be replaced with a dialog that lists all the functions in that module. Those on the left have been selected, those on the right have not. Double-clicking on a function name moves it to the other column. When the functions you want to probe from the selected module are listed on the left, click Ok, and list in the configuration gui itself will be updated.

Note: this process can be very slow for large applications with many thousands of symbols (see [#MARKER-9-1931 "Performance Issues"]). In such cases it may be easier to use [aprobe-9.html#MARKER-9-1016 apcgen] or [aprobe-9.html#MARKER-9-1052 apinfo] to generate symbol lists, and edit those lists by preceding them with the keyword. For example:

apsymbols main | sed "s/^/Trace / " >> main.trace.cfg

Note that for memwatch.ual, selecting individual functions isn't necessary. Instead the MemWatch probe displays graphs at run time showing memory usage patterns, and allowing interactive snapshots.

Callable API

You can invoke functions defined by a predefined probe from within your own probes. The interface is defined by the header file in the [../include/ $APROBE/include] directory whose name corresponds to the predefined probe; for example, trace.h is the interface for trace.ual.

The most useful functions are generally at the end of the file: those which enable and disable the probes, and those which log a "snapshot" of the data currently selected. See the comments in each file for details.

Note that there is no info.h file -- its operation is entirely determined by its command-line parameters; and the quick_gui.h file is the only interface to quick_gui.ual, since it defines no probes.

Snapshots

Trace, profile, coverage, statprof and memwatch all provide the ability to take a "snapshot" of the data at a point during a program's execution. A snapshot is simply a dump of the data collected in memory since the start of execution or the previous snapshot. The coverage, memwatch, and trace probes support the keyword SNAPSHOT in the configuration file, with which a function name is specified. For example, in your main.trace.cfg file you would do:

Snapshot extern:"LOCAL_PROC()" On Exit

This is translated into a probe which calls a function to take the snapshot. You can include this in your own probe--call it mytrace.apc--as follows:

 #include "trace.h"
 probe thread {
   probe extern:"LOCAL_PROC()" {
     on_exit
       ap_Trace_DoSnapshot("after LOCAL_PROC");
   }
 }
 Then, compile this probe, using trace.ual as a library:

apc mytrace.apc trace.ual -o mytrace.ual

and then use  mytrace.ual instead of trace.ual to gather data:

aprobe -u mytrace.ual main

Example D-1. Calling ap_Trace_DoSnapshot()

Info Probe: info.ual

The info.ual predefined probe provides the user with the general information available about the program. It can be used on an executable before the program is run, or with apformat to see additional information about the program after it was run.

The most common use of this probe is to get the names of the target program's entry points for which one can define a probe. Without using this probe, it is sometimes difficult to come up with the exact name of the function, that would be understood by Aprobe. Using this probe ensures that the one refers to target functions by the same name as does Aprobe.

Using info.ual directly with aprobe is made unnecessary by the apinfo and apsymbols commands (see [aprobe-9.html#MARKER-9-1052 "apinfo"] and [aprobe-9.html#MARKER-9-1110 "apsymbols"]). These are scripts which pass invoke aprobe using info.ual and simplify the handling of the parameters and other files.

Another important use of this probe is at format time to get the names of all the [aprobe-10.html#MARKER-9-1336 UAL file]s that were used to collect the data at runtime. This is especially useful in lab environments where the multiple users may be collecting different data using different UALs on the same executable at the same time.

Usage

info.ual is specified on the [aprobe-9.html#MARKER-9-1075 aprobe] or [aprobe-9.html#MARKER-9-1035 apformat] command line. The specific options are:

aprobe -u info.ual [-p " param_list "] executable apformat -u info.ual [-p  " param_list " ] apd_file

where param_list may include:

-a

Prints all information.

-d

Print data symbols from application module.

-da

Print data symbols from all modules.

-h

Prints options.

-l

Prints instrumentable lines for each function symbol shown.

-m

Prints the list of modules and their checksums.

-s

Prints function names from the application module.

-sa

Prints function names from all modules.

-t

Prints the list of threads (format-time only)

-u

Prints the list of UALs used at runtime (format-time only)

'-x'

Indicate functions that are not instrumentable.

'-xb'

Exclude listing functions that are not instrumentable.

'-xg'

Exclude listing functions that are instrumentable.

For example, the command

  aprobe -u info.ual -p "-s" a.out

Provides the same output as the command "apsymbols a.out".

If one doesn't know what UALs were used in data collection at runtime, he can get this information with the following command:

  apformat -u info.ual -p "-u" a.apd

quick_gui Library

quick_gui.ual is a library of functions to provide simple Java GUI support to a user-written probe. The interface is defined in [../include/quick_gui.h $APROBE/include/quick_gui.h], and includes histogram and plot graph support, as well pop-up Yes/No and message dialogs. Use of this library is illustrated by the [../examples/learn/visualize_data/README $APROBE/examples/learn/visualize_data] example.

The contents of quick_gui.h is reproduced here with some additional comments and an example.

Histogram

For each monitored variable in your program, call ap_CreateHistogram() to allocate an object that will keep track of its values' frequency distribution.

typedef struct _ap_HistogramObject_ *ap_HistogramObjectPtrT;
 
 ap_HistogramObjectPtrT ap_CreateHistogram(
      ap_NameT Title,
      int      Granularity);

void ap_UpdateHistogram(
      ap_HistogramObjectPtrT Histogram,
      int                    NewValue);

Call ap_UpdateHistogram() with the current value of the monitored variable.This can be done periodically, or every time the variable changes.

ap_UpdateHistogramValues() will add a new value to the histogram, but will not redraw it. Use it when you want to display a large number of updates in one batch operation.

void ap_UpdateHistogramValues(
      ap_HistogramObjectPtrT Histogram,
      int                    NewValue);

Call ap_DisplayHistogram() to redraw the histogram with the current values. This can be done periodically (using ap_DoPeriodically()), while ap_UpdateHistogramValues() could be called for each new value of the monitored variable.

void ap_DisplayHistogram(
      ap_HistogramObjectPtrT Histogram);

XY Graph

This is a traditional XY graph, such as used by the Heap Probe. To use it, you create a graph object with a few basic attributes, then send points to be plotted on the graph.

Call ap_CreateXYGraph() to creates a new empty XY graph object. Provide the title and the labels for the axes. \

NumberOfPoints indicates the maximum number of pairs kept in the graph, before the data scroll off the left side of the graph. Give zero to keep all the data points and compress the X axis instead of scrolling.

typedef struct _ap_XYGraphObject_ *ap_XYGraphObjectPtrT;
 
 ap_XYGraphObjectPtrT ap_CreateXYGraph(
      ap_NameT Title,
      ap_NameT X_AxisName,
      ap_NameT Y_AxisName,
      int      MaxNumberOfPoints);

Call ap_PlotGraphUpdate() with the current X and Y coordinates.

void ap_UpdateXYGraph(
   ap_XYGraphObjectPtrT Graph,
   double               X,
   double               Y);

Yes-No Dialog

ap_AskYesOrNo() is a simple call you can make to prompt the user before taking some action. Returns TRUE if Yes, FALSE otherwise.

ap_BooleanT ap_AskYesOrNo(
   ap_NameT QuestionToAsk,
   ap_NameT Title,
   ap_NameT HelpText);

Message Dialog

ap_DisplayMessageAndWait() simply pops up a dialog containing the message provided, and waits for the user to click OK.

void ap_DisplayMessageAndWait(
   ap_NameT Title,
   ap_NameT Message,
   ap_NameT HelpText);

Text Input Dialog

Call ap_InputText() to prompt the user for a text string, such as a name or value, and wait for the user to click OK or Cancel. Returns TRUE and the InputTextBuffer if the user chose OK; returns FALSE if Cancel was pressed.

ap_BooleanT ap_InputText(
   ap_NameT Title,
   ap_NameT Prompt,
   ap_NameT HelpText,
   char     *InputTextBuffer);

Example

The following is "visualize_data.apc" found on-line with the corresponding test program in [../examples/learn/visualize_data/README $APROBE/examples/learn/visualize_data]. This probe will graphically display the value of the target variable MyTargetVariable by plotting its value every second and by drawing the histogram of the frequency distribution for each value of it.

The first part shows the declaration of the graph objects and the function that is called to update them. The second is the "probe program on entry" action that initializes the graph objects and registers the update function to be called periodically.

 #include "quick_gui.h"
 
 static ap_HistogramObjectPtrT HistogramObjectPtr = NULL;
 static ap_XYGraphObjectPtrT   XYGraphObjectPtr   = NULL;
 
 // Define the action that updates the graph, to be called by
 //ap_DoPeriodically()

void MyPeriodicAction(void *EP)
{
  // Note the use of the target expression $MyTargetVariable.
  // MyTargetVariable is not declared in this file, but rather is the name
  // of a variable declared in the target program.

 ap_UpdateHistogram(
    HistogramObjectPtr,           // Pointer to the histogram object
    $MyTargetVariable);           // Integer value to update the histogram with

  ap_UpdateXYGraph(
    XYGraphObjectPtr, // Pointer to the XYGraph object
    ap_TimeToSeconds(ap_ElapsedTime()),
                // Seconds since the program start
    $MyTargetVariable);           // vs. the value of MyTargetVariable
}

Example D-13a. Define Graph Objects and Update Function

probe program
{
  on_entry
  {
    // Create a new histogram object that could be used in updates
    HistogramObjectPtr = ap_CreateHistogram(
      "My Histogram",         // Title of the histogram
      5);   // Granularity of the histogram

    // Create a new XYGraph object that could be used for plotting
    XYGraphObjectPtr = ap_CreateXYGraph(
      "My Plot",        // Title of the Graph
      "Time",       // X coordinate name
      "Value of 'MyTargetVariable'",
              // Y coordinate name
      0);       // Number of points, 0 => keep them all

    // Register to update the graphs every second.
    ap_DoPeriodically(
      MyPeriodicAction,           // Periodic action to perform
      1,            // Do it every 1 second
      NULL);            // No data to pass to the periodic routine
   }
}

Example D-13b. Create Graphs and Register Update Function

---

Copyright 2006-2017 OC Systems, Inc.