AUG Predefined Probes

From OC Systems Wiki!
Revision as of 23:06, 27 February 2019 by Swn (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Predefined Probes and Libraries Reference


The Aprobe product includes ready-to-use UALs built from probes written by OC Systems. We call these "predefined probes". These may be referenced by name on the aprobe command line and applied to any application, without writing any APC or using the apc command.

This appendix provides general information about how the major predefined probes work and how to use them.

See Aprobe Predefined Probes for a list of all the predefined probes and in-depth information about each probe.

In most of this appendix, we'll refer to these by their simple filename: coverage, events, profile, trace, memwatch, etc.

How They Work

The predefined probes are written using the features and interfaces described in this user's guide, and could have been written by a user. As such, they demonstrate the power and flexibility of Aprobe. But make no mistake--they are complex, and are not recommended as models or examples for learning Aprobe. Nonetheless, it is useful to understand what happens when you apply one of these probes, particularly as it relates to performance of your application and your choice of options.

When aprobe is invoked on an application, it loads the executable, the libraries it references, and the specified UALs and reads in all the symbol table information. The aprobe runtime applies all the statically defined probes described in the UALs, then invokes any probe program on_entry actions. It is in such a probe that all the start-up actions for a predefined probe occur: UAL command-line arguments are read; the configuration file is read and recorded; and, for those probes that support it, the Java GUI process is started if requested. Using the GUI, the user may request a list of all the symbols in a module from which to select those to be probed, and specify other options.

When the final configuration is set and execution is continued, the probes on the functions specified in the configuration file are applied. When all probes have been applied successfully, and any other probe program and probe thread on_entry actions have been executed, the application program continues its normal execution.

As the application executes, the probes are invoked and collect or update the data according to their function. If snapshots are used, the data is written to the APD file at that point -- otherwise the data is written out when execution finishes.

When execution is completed, the APD files created by the probe contain the data collected. Using apformat on these produces the data collected which of course varies with each probe.

Performance Issues

As the application executes, there may be thousands of different probes, executed millions of times, all collecting data. The memwatch probe will only probe a few system functions that manipulate heap memory, but these may be invoked thousands of times. The coverage, profile, and trace probes are invoked for every function (and perhaps every line of every function) that was specified in the probe's configuration file. If data is logged directly to an APD file on disk (as the trace probe can do), this may be millions of bytes of I/O being written. If data is collected in memory, as is done by the memwatch probe, a long-running or large program may require a lot of virtual memory.

The areas where performance of a probe is noticed are:

  1. Aprobe (and apformat) start-up, when the symbol tables are read from the target application and its libraries;
  2. The Configuration GUI, when lists of symbols must be built for display and manipulated by the interface;
  3. Instrumentation, when probes are patched into the entry, exit, and perhaps each line of the user-selected list of functions.
  4. Execution, when the probes are invoked and executed as an extension to the user's program;
  5. Data collection, when the probes record their data in memory, and perhaps write it to disk; and
  6. Data formatting, when the data that has been logged is analyzed.

The perceived speed of items (1) and (2) is directly related to the size of the application, or more specifically, the number of symbols in the application. The instrumentation time (3) is directly proportional to the number of probes being applied, which for the coverage, profile, and trace probes means the number of functions selected in the configuration file or GUI. The instrumentation time for the memwatch and statprof probes is constant and fast. The execution overhead (4) of the probes once they're started depends on the number of probes actually invoked, which is a product of the number of functions selected by the user and the number of times each such function is called. The data collection and formatting times (5, 6) vary greatly for each probe, and are described in the Performance Issues section for each probe. In brief, however:

  • The memwatch probe records data for each memory allocation, proportional to the amount of memory allocated by the application under test.
  • The coverage probe simply increments an integer for each line executed, so the amount of data is a constant multiple of the number of lines being covered.
  • The profile probe simply adds to a total execution time kept for each function, so the data is proportional to the number of functions being profiled.
  • The trace probe may be very data intensive at run-time unless load-shedding is enabled, or the fixed-sized circular buffer log method is specified, since every entry and exit (and optionally every line) records many bytes of data.
  • The events probe is similar to the trace probe in overhead. In fact trace may be used in conjunction with events in order to provide load-shedding, even if nothing was traced.

Common Interfaces

The behavior of a predefined probe may be controlled in a number of ways:

  • by passing parameters to the UAL on the aprobe command line;
  • by editing a text "configuration file"
  • by interacting with a Java Graphical User Interface (GUI) controlled by the probe;
  • by providing your own probe which explicitly calls functions exported from the predefined probe.

These mechanisms are much the same for the dynamic analysis probes above.The info.ual predefined probe is a utility for which all behavior is specified on the command-line; and quick_gui.ual is a library, so all interaction with it is by function calls from within a user-written probe. The statprof.ual predefined is another simple probe that does not require a configuration file or GUI to customize its behavior.

Command Line

In the on-line "4.predefined" example, a command line like the following is used:

aprobe -u trace.ual -p "-g -v" main

The trace.ual file is actually located in the directory $APROBE/ual_lib, but this directory searched by default for any UAL not found in the current working directory, so you only need to provide the base name of the UAL file.

The -p option which follows trace.ual indicates that the next argument is a string of options to be passed to the UAL. In this case, those options are -g, which indicates that the Java Graphical User Interface (GUI) is to be started; and -v, which causes "verbose" informative output to be generated by the probe. Every predefined probe accepts the option -h, which describes the full set of command-line options available to that probe. So, to display the full set of options for trace.ual, execute:

aprobe -u trace.ual -p -h main

Predefined probes are generally initiated by invoking aprobe explicitly. However, they may also be invoked implicitly by linking your application with or using the run_with_aprobe_apo script and specifying the command-line options in a .apo file as described in Loading Probes Without aprobe.

Configuration File

The configuration file is used to designate which functions in the target application are to be analyzed, and to specify other options. The easiest way to get an initial configuration file is simply to invoke the predefined probe with no options. You'll get a warning about not finding a configuration file, but one will be created.

You can use the Congifuration GUI to create or change the configuration file by specifying the -g option. This provides a point-and-click interface in which to choose your functions and other options and save them in a configuration file. (Note that this GUI can be quite slow for very large applications, so editing the file directly may be more practical in such cases.)

The default name for the probe configuration file is your_program.probe_name.cfg, where your_program stands for the name of your executable application program, and probe_name is the basename of the predefined probe, e.g., trace. For example, if your program is called wilbur.exe and you're using trace.ual, then the default configuration file name would be wilbur.exe.trace.cfg.

The format of the a probe configuration file is:

keyword data
keyword data
// This is a comment line

A warning will be given if the very first line of this file isn't valid, but an attempt will be made to use the information provided. The only allowed variations on this header line are the number of blank spaces between the words, and the case of each character.

All keywords in the configuration file are case-insensitive. However, all function or module names are always case-sensitive.

Each line begins with a keyword, or is a continuation line. A line is a continuation line if the previous line ends in a backslash ("\"). The continuation character is treated as a word separator, like a blank.

Comment lines can be interspersed with any other lines. Comment lines are those lines whose first 2 non-blank characters are "//". Comment lines are ignored by probe and are intended only for the benefit of the human reader.

Any invalid line is ignored. An invalid line is one which contains an invalid keyword or value, or has a missing or extraneous word.

Configuration of Selected Functions

The coverage, profile, and trace predefined probes operate by probing functions in the user's application, so the names of those functions must be specified in the configuration file. While each predefined probe uses different keywords to designate which functions to probe, the syntax of the function name itself is the same in all cases. It is exactly the same as the syntax used for a probe itself, as described in Specifying Function Names and repeated here:

[ [file_name | extern] : ] function_name [ in module_name ]

file_name, function_name, and module_name are all strings that are bounded by quotation marks (that is, ""), which allows each name to contain embedded blank space characters. The file_name and the module_name are optional. If module_name is omitted, the application module is assumed, and the special character "*" can be used to designated every module. The file_name is required only for static (file-local) functions. A colon must exist between the file_name and the function_name. The colon may be surrounded by optional whitespace, and the keyword IN must always be surrounded by whitespace. Examples are:

extern:"malloc()" in ""

In the configuration file for trace.ual, the keyword indicating that a function is to be traced is "Trace", where the cases of the letters is not significant. In the main.trace.cfg file, then should appear one or more lines that look like

Trace "my_file":"my_function"

Predefined probes may have other keywords, such as Trigger, Remove and Snapshot, which are followed by function names. These are described in their corresponding section below. And, note that memwatch.ual does not require specifying application functions this way -- see memwatch.ual.

Configuration GUI

Some, but not all, predefined probes offer a Configuration GUI (Graphical User Interface). The coverage, profile, and trace predefined probes do offer it, but not events, memwatch, nor info

If the -g option is specified to the UAL, the probe starts a Java process to run the Configuration GUI. This provides a point-and-click interface for selecting functions and setting other options which are recorded in the Configuration File. This interface differs for each probe, but generally consists of several panes in which functions and option settings are selected. At the bottom are Save & Run , Run, Save As, Abort, and Help buttons. Run proceeds with the execution of the program under aprobe with the current configuration; Save updates the configuration file on disk with the current settings; Abort quits the GUI and aborts the aprobe and target application processes; and Help brings up a dialog describing each pane and the other buttons.

Function Selection Interface

The Configuration GUI can be useful for selecting the functions that are to be operated upon by the probe. The "4.predefined" example illustrates this for the trace probe, and this process generally applies to the other probes as well:

Click on the "Pick..." button under the top pane (for Trace, the top pane is labeled as "Traced Functions"). You'll be presented with a list of modules (the executable plus any shared libraries it loads). When you double-click on one of those, it will be replaced with a dialog that lists all the functions in that module. Those on the left have been selected, those on the right have not. Double-clicking on a function name moves it to the other column. When the functions you want to probe from the selected module are listed on the left, click Ok, and list in the configuration gui itself will be updated.

Note: this process can be very slow for large applications with many thousands of symbols (see Performance Issues). In such cases it may be easier to use apcgen or apinfo to generate symbol lists, and edit those lists by preceding them with the keyword. For example:

apsymbols main | sed "s/^/Trace / " >> main.trace.cfg

Note that for memwatch.ual, selecting individual functions isn't necessary. Instead the MemWatch probe displays graphs at run time showing memory usage patterns, and allowing interactive snapshots.

Callable API

You can invoke functions defined by a predefined probe from within your own probes. The interface is defined by the header file in the [../include/ $APROBE/include] directory whose name corresponds to the predefined probe; for example, trace.h is the interface for trace.ual.

The most useful functions are generally at the end of the file: those which enable and disable the probes, and those which log a "snapshot" of the data currently selected. See the comments in each file for details.

Note that there is no info.h file -- its operation is entirely determined by its command-line parameters; and the quick_gui.h file is the only interface to quick_gui.ual, since it defines no probes.


Trace, profile, coverage, statprof and memwatch all provide the ability to take a "snapshot" of the data at a point during a program's execution. A snapshot is simply a dump of the data collected in memory since the start of execution or the previous snapshot. The coverage, memwatch, and trace probes support the keyword SNAPSHOT in the configuration file, with which a function name is specified. For example, in your main.trace.cfg file you would do:

Snapshot extern:"LOCAL_PROC()" On Exit

This is translated into a probe which calls a function to take the snapshot. You can include this in your own probe--call it mytrace.apc--as follows:

#include "trace.h"
probe thread {
  probe extern:"LOCAL_PROC()" {
      ap_Trace_DoSnapshot("after LOCAL_PROC");

Then, compile this probe, using </nowiki>trace.ual as a library:

apc mytrace.apc trace.ual -o mytrace.ual

and then use mytrace.ual instead of trace.ual to gather data:

aprobe -u mytrace.ual main

Example D-1. Calling ap_Trace_DoSnapshot()

[Next] [Previous] [Top] [Contents] [Index]

Copyright 2006-2017 OC Systems, Inc.