Difference between revisions of "AUG 3 Writing APC Probes"

From OC Systems Wiki!
Jump to: navigation, search
m (Using a debugger on a stand-alone probed executable.)
m
Line 34: Line 34:
 
* [[#on_line|on_line]]
 
* [[#on_line|on_line]]
 
* [[#on_offset|on_offset]]
 
* [[#on_offset|on_offset]]
* [[#log|og]]
+
* [[#log|log]]
 
* [[#Target Expressions|target expressions]]
 
* [[#Target Expressions|target expressions]]
  

Revision as of 16:53, 20 February 2019

Contents

Writing APC Probes


Introduction

This chapter describes how to write probes in for native programs in APC. Probes for Java programs are written in Java, and are described in Chapter 5, Writing Java Probes.

When you compile a probe using the apc command, three things happen:

  1. The APC file(s) are processed through the Aprobe preprocessor. Each Aprobe directive in your probe source causes various expansions and calls to Aprobe runtime routines.
  2. The processed file(s) are compiled with your target C compiler and object file(s) are produced.
  3. The object files produced in step 2 and any other object files that your code might need are linked to produce a shared library. We call this library a "User Action Library" or UAL.

Generally you can think of this as a single compilation step, that produces a UAL from APC files. However, you could, in theory, omit step 1 and write C code directly, calling the correct runtime routines and such. You can view the output of step 1 if desired: Use the apc -C option to generate this file. Note that this file is normally destroyed after it is compiled and so is not formatted for human readers.

The rest of this chapter explains the most commonly used Aprobe directives. A complete list can be found in APC file.

The Aprobe Preprocessor Directives

You have already seen most of the directives explained in this chapter, because they were used in the example in Chapter 2. We will be discussing:

The Aprobe preprocessor directives use C syntax and allow you to easily specify the locations to be patched and the data to be logged, as well as the format functions you wish to use for printing out the logged data. All of the directives use source level names, making it simple to quickly write robust probes.

The syntax and semantics of the Aprobe preprocessor directives are fully defined in APC file. The goal here is to illustrate the structure of a probe and describe the most useful constructs. This description, combined with the on-line examples and some experimentation should enable you to do nearly everything you might want to do. If not, there's Advanced Topics.

The probe Directive

The probe directive specifies the entity (usually a subprogram) to be probed. It defines the context in which the subsequent directives execute.The legal arguments to the probe directive are: keywords program, thread, format, all, or else the quoted name of a function.

When the target program starts execution, the C code in the on_entry section of a probe program gets executed. Similarly, when the target executable program terminates, the C code in the on_exit section of a probe program gets executed. Each time a new thread comes into existence, the C code in the on_entry section of a probe thread gets executed. Similarly, the C code in the on_exit section of a probe thread gets executed whenever any thread terminates.

When a function is entered (or exited), the C code in the on_entry (or on_exit, respectively) section of the probe "function_name" for that function is executed. The C code for the on_line sections of the probe is executed just before the first instruction of the specified line is executed.

The following APC file summarizes the structure of these probes, even though this example contains only comments within the probes themselves.

probe program
{
  on_entry { /* program startup, before calling main() */ }
  /* on_line doesn't make sense here */
  on_exit  { /* program normal exit or via exit() */ }
}  /* end of probe program */

probe thread
{
  probe "RoomClass::GetSize" {
    on_entry { /*before RoomClass::GetSize starts*/ }
    on_line(13, 117..155) { /* for each listed line */}
    on_exit  { /* before returning from GetSize */ }
  }  /* end probe on GetSize */

  on_entry { /* thread creation, always at least one */}
  /* on_line doesn't make sense here */
  on_exit  { /* thread termination */ }

}   /* end of probe thread */


Example 3-1. Program, thread, and function probes

The "on" directives (e.g., on_entry) are described in the next section, [[#Probe Action Directives|Probe Action Directives]. See the Example of Probe Nesting and Visibility for an example of how data is visible in different parts of a probe.

Note that the actions described using the probe directives (i.e., probe program, probe thread and probe "function_name") occur while the program is running. These actions can read, change or log data. After the program has finished running, you use apformat to format the raw data with associated format subprograms (see Logging and Formatting Data).

Aprobe has some very powerful formatting and data reduction capabilities in addition to the automatic formatting that is available via the log directive.The C code in the on_entry section of a probe format gets executed before the apformat tool starts reading any of the logged data from the APD file or other logging device, allowing for the initialization of user data structures or even a graphical user interface. Similarly, the C code in the on_exit section of a probe format gets executed after all of the logged data has been processed, so that results may be summarized after all data has been formatted.

Interaction Between Program and Format Probes

As described in detail in Logging and Formatting Data there can be two distinct phases of processing with aprobe: run time, when the data is collected using aprobe, and format time, when the collected data is formatted for reading using apformat.

Normally, probe program is executed at run time, and probe format is executed at format time, and their data spaces are separate. However, if you run aprobe with the -if (immediate format) option, both the probe program and probe format probes are executed at run time. In particular, the on_entry part of the probe format is executed first; then the on_entry part of the probe program, then the on_entry part of the main thread probe. At program end, the on_exit part of the probe format is executed last.

This is of concern when the format probe and other probes access the same data. If you have data that will be overwritten when aprobe -if is used, you should bracket the initialization with:

if (ap_CurrentAprobeState() == ap_AprobeFormatTime)
{
  // This initialization is only performed when it's truly
  // format time, not when we are immediately formatting
  // at run-time.
}

Specifying Function Names

To probe a particular function, use the probe directive and specify the function's name. Function names are specified using a special syntax, which helps to differentiate between multiple functions that have similar though distinct names. This syntax is designed so that a full specification is always unique, but a partial, more natural name is often sufficient.

Most compilers transform such function names into uniquely encoded names, a process known as mangling. Mangling styles differ from compiler to compiler, but share two properties -- they are unique across the whole program so the symbolic linker may do its job, and they have encoded in them sufficient information to produce a human-readable name similar to the one in the source code. The translation of the mangled name to the human-readable name is called demangling. Debuggers (e.g., dbx, gdb) perform this demangling, but do not have a requirement that the human-readable (demangled) name remain unique, since the user can interactively select the right one. Aprobe does have such a uniqueness requirement, but attempts to demangle names in a manner similar to that used by the native debuggers, (and to how the user expects them to look), while maintaining uniqueness.

In general, it is only necessary to specify enough of the demangled name for it to be uniquely identified in the symbol table of the executable being probed. At preprocessing time, apc will expand a partial function name to its fully qualified name. If the name is ambiguous (there is more than one possible match for the name you specified), it will choose the first and give a warning, listing all the possible matches.

If you specify a name that must be resolved at runtime--such as when you name a function in the configuration file of a predefined probe--you should provide the unique, fully-qualified name for each function, because the first partial match will be selected. Examples of names are given below.

To determine the full name of a function in an executable, use apcgen or apsymbols.

"main"        // partial, but usually enough

"foo()"       // means you want foo, not Class::foo

"foo.c":"foo" // the static function in foo.c

"EmployeeClass::FindEmployee"
        // may match several overloaded names

extern:"EmployeeClass::FindEmployee(char*)"
        // specifies exactly one, which is global

"main.main"       // The Ada library subprogram called main

"ada.text_io.put_line[2]"
        // 2nd overloading of put_line

Example 3-2. Probe function names

C function and method names

Overloaded methods are resolved by explicitly specifying the parameter profile. The profile must match precisely with what has been recorded by the C compiler, so it is usually necessary to pick the one you want from a list. In addition to using apcgen or apsymbols as mentioned above, the command "nm -C exe_name" will display the full names of functions in the module named "exe_name".

Ada subprogram names

The apc compiler provides support for the GNAT Ada compiler on Solaris and Linux, and the PowerAda Ada compiler on Linux and AIX. In a function probe, apc recognizes Ada subprogram names like: "package_name.subprogram.nested_subprogram", and attempts to resolve partially qualified names like "nested_subprogram".

Overloaded names in the same scope are resolved by putting "[n]" after the name. For example, ada.text_io.put_line[1].

For a library subprogram, such as the main subprogram, one can generally use the name as it appears in the source. If there is a conflict with a C function of the same name, (for example, if the Ada main subprogram is called "main"), one simply repeats the name of the library subprogram as if it were a package name: main.main.

Probe Action Directives

The "on" directives: on_entry, on_exit and on_line are the probe action directives. Each is translated into a function that is invoked at the specified point in the program with parameters describing its context both relative to other probes and within the probed executable (see Parameters to Probe Actions).

Probe action directives must be nested within a probe directive. The probe directive provides the context for the subsequent directives. For example, when you log a data item from the probed executable program (by using the $ in front of the item's name), the Aprobe preprocessor can determine the type (and hence the format) for the data value because the log statement is within the context of the probe.

The C code executed for each probe action directive may be any form of C statement, including a compound statement, or it may be any form of C declaration. This means it may declare additional variables or call any available subprograms, including subprograms defined in the user's original executable program. Variables declared in a compound statement exist only for that compound statement.

All probe action directives are cumulative rather than replacing previous ones. This means that, for example, all of the multiple on_entry statements will be executed in lexical order, rather than executing only the first or last on_entry statement. Similarly, every probe on a given function will be executed, even if there were multiple such probes defined for the same function.

on_entry

For function probes, the on_entry directive specifies that the associated C code is to be executed before the probed function is entered. For program and thread probes, the C code statements of the on_entry section are executed when the program or thread is about to start. For format probes, the on_entry section's C code is to be executed before the formatting of the first logged item.

The on_entry action for a function probe is the most common probe, since parameters to the function may be reliably examined and modified at that point as described in Target Expressions. Also, ap_StubRoutine is applied on an on_entry action to prevent further execution of the function (see Stub Support).

It is important to note that a function's automatic local data from the target program is not visible from an on_entry action. That's because the function's stack frame (in which the data will be declared) has not yet been pushed. However, any "static" function data, and data declared in enclosing Ada scopes, is visible from all actions within a function because it's not local to the function's stack frame.

on_exit

For function probes, the on_exit directive specifies that the associated C code is to be executed just after the function has exited, but before the probed function returns to its caller, whether by return statement, by exception propagation, or by some other mechanism such as a call to longjmp(). For thread and program probes, the C code statements of the on_exit section are executed upon the thread's termination or the program's termination, respectively. Similarly, for format probes, it is executed after formatting the last logged item.

The on_exit action for a function probe is where you may examine or modify either the return value of a function (via $0 or $return) or any reference parameters that may have been updated by the function (see Target Expressions).

It is important to note that, as with on_entry, automatic local data is not visible from on_exit, since the stack frame in which the function's local data was declared has already been discarded.

Scalar parameters are not visible on_exit, since they are stored in the same place as local data upon entry to the function.

on_line

The on_line directive takes as an argument a source line number within the enclosing probed function. It specifies that the following C code is to be executed just before the first instruction of the specified line is executed. The on_line directive may not appear in a probe program, a probe thread, a probe format or a probe all.

The on_line directive may only be used in a function probe if line information is available for the function, which means it must be compiled with debug (-g).

The execution of an on_line directive, and the querying of data from within that action, is similar to a breakpoint in an interactive debugger, and as such is subject to the re-ordering of instructions and caching of data by the compiler. In particular, an on_line action may not occur as expected if the line is in optimized code.

The "source line" argument to the on_line directive is actually very flexible:

  • it can be the keyword all;
  • it can be a single line number, a comma-separated list, a range of line numbers, or any combination of these;
  • a "single line number" may be a positive integer constant, the keyword first, the keyword last, or a constant expression consisting of a combination of these.
  • For example on_line(first 5) allows a relative line number to be probed without the possible problems introduced by changing the code surrounding the probed function.

Except for on_line(all), which is evaluated at aprobe run-time, the on_line directive can be compiled only if all of the following are true:

  • an executable file is provided to apc
  • the function being probed has debug information associated with it;
  • the line number specified is explicitly identified in the debug information as a "probe-able" line.

The apc compiler will generally give an error message explaining this. You can determine what lines may be probed, as well as which line is the first and last, by using apcgen with the -qlines flag. For example, if you want to probe function "foo", in executable "bar.exe" you can generate a probe on function foo that includes all possible on_line directives with the command:


apcgen -p "foo" -qlines bar.exe

As mentioned above, local target variables are generally only visible from an on_line action, since on_entry and on_exit actions are executed in the scope of the caller, before and after the local data has been allocated. If local data is defined directly within the function's opening brace (or after the Ada reserved word "is"), then it should be visible from on_line(first) and on_line(last). But note that, because the action is executed before the specified line, the data will not be initialized at on_line(first), even though this would have been the ideal place for a probe to initialize the data.

on_offset

The on_offset directive takes as an argument a numerical byte offset from the start of the enclosing probed function. This is useful for probing assembly-language functions as well as those for which no debug information is available, or which have been optimized. You must determine the correct offset using the native disassembler or debugger. As with on_line, the on_offset may not appear in a probe program, a probe thread, a probe format or a probe all.

The log Directive

The log directive starts a log statement, which looks somewhat like a function call and specifies that the value or values given as arguments are to be "logged" (written as raw data) to a data file (the APD file) for later formatting. The log statement collects the data, and also and indicator of the function to be used to format the data. This formatting function may be designated by the user or may be left unspecified and generated automatically by the APC compiler. A more complete description is given in Logging and Formatting Data.

Target Expressions

Conceptually there are two name spaces: the application or target name space and the probe name space. We call expressions that use the application name space target expressions, since we use the word "target" to refer to the application. The Aprobe preprocessor uses a $ (dollar sign) to differentiate between the two name spaces. Aprobe makes it simple to reference identifiers (e.g., data, functions) from the application name space by merely placing a $ in front of the identifier. Without a $, the preprocessor assumes that the identifier is defined in your probe's name space.

A target expression, then, is an expression (perhaps just a variable or functionname), denoted by a leading $. This mechanism is used to denote low-level values, such as registers and positional parameters, as well as actual source-level identifiers and functions from the user's program.

Target expressions allow probes to have full access to the data and environment of the probed application. They are expanded by the preprocessor to do the machine level addressing (registers, offsets, type-casting) necessary to refer to the objects when the probe executes.

"Target expressions" shows some examples of target expressions. The complete syntax is described in APC file.

$$i0      // sparc target register '$i0'
$$r3      // PowerPC target register '$r3'
$$EAX     // x86 register '$EAX'

$a      // program variable or parameter 'a'

$1      // the first positional parameter

$*      // all positional parameters, with names

$0      // the function return value (on_exit only)
$return      // also the function return value

$("X", "-file coord.c")   // static variable X in 'coord.c'

$("P", "-unit lib/pkg")   // static variable P in PowerAda unit lib/pkg

typeof($PositionType)   // the type $PositionType

$UserFunction(var1, 7); // call 'UserFunction' in application
                        // with two parameters.

$("ListClass::DebugDump(char*)")($this, "ProbeDump");
      // a method call to dump the list object

$("Globals.Item_List").length

$("List_Ops.Remove_Head")($("Globals.Item_List"));

Example 3-3. Target expressions

A machine register is designated by a target-specific name preceded by double dollar sign, e.g., $$EAX.

Target expressions like $1, $2, etc., refer to the parameters passed to the subprogram being probed.

The target expressions $return and $0 both refer to the return value of a function and are valid only in an on_exit section.

The target expression $a refers to a data item named a.

The target expression $* refers to all input parameters when $* is used in an on_entry or on_line section and to the return value when $* is used in an on_exit section. The $* may be used only in a log statement with automatic formatting.

Local data items are visible only from on_line sections. In particular they are not visible on_exit because the stack frame is no longer available. To evaluate local data items as close as possible to the exit of the function, use on_line(last).

Sometimes a simple name is not sufficient to denote an entity, so the name must be enclosed in quotes, and/or additional context information supplied. In this case, the general form of a target expression must be used:

$("anything goes", "myfunc() -file file.cpp -module mylib.so")

The second value in quotes in the expression above is a "context_description". The components of a context_description are used to resolve names in the case of collisions or ambiguity. If a context_description is not provided, the name must be visible and uniquely determineable from the point of the probe. The syntax of a context_description is:

"[ function_name ] [ -module module_basename ] [ -file source_file_basename | -unit powerada_unit_name ]"

The function_name must appear first, if present, and is assumed to be everything up to the end of the string or the first -file, -unit or -module keyword.

The module_basename is the filename (without directory path) of an executable or shared library that was provided on the apc command line. This name is necessary so it can be saved in the UAL, allowing aprobe to correctly resolve the expression at runtime.

The source_file_basename is used to distinguish between static names (or names visible only within a body) which may conflict with global names or may appear in multiple source files.

The powered_unit_name is used to designate a PowerAda unit name (e.g. "lib/pkga" or "sec/pkga")..

An expression may consist of a function call. However, only the identifier preceded by a $ is treated as a target expression; the actual parameters are resolved in the Aprobe name space unless they are also target expressions. Hence to call a user's program's function, one could write something like:

$UserFunction(var1, 7);

which means that $UserFunction is resolved (or demangled) based on the application name space (i.e., the executable file and its shared libraries) provided to the apc command, and var1 is resolved in the probe name space.

In the call:

$("List_Ops.Remove_Head")($("Globals.Item_List"));

the form is function_name ( argument ) where the function_expression is
$("List_Ops.Remove_Head") and the parameter is
$("Globals.Item_List").

Resolution of target expressions requires debug information in an object module, which is put there by the target compiler. A general rule is that if your target debugger can print out a variable by its name, then the apc preprocessor can resolve it as well.

It is worth noting that positional parameter target expressions do not require debug information or even an executable in order for their locations to be resolved. That is, even if you have an undebuggable executable, you can usually use $1 to access the first parameter to a function and $0 or $return to access the return value. In such cases the user must cast the parameter to a C type defined within the APC file, or supply an explicit format routine, because the type of the parameter or return value is not known (see User-Supplied Formatting). Float, struct, and 64-bit parameters are often passed differently and may require direct reference to the hardware registers.

Because target expressions reference names in the target program, and a target program may be written in several different languages, you might wonder how to reference constructs specific to that language.

The general rule is: you must understand how that language implements that construct relative to C, and expressions must be given using C syntax. Target entity names (but not operations) may use non-C syntax if they appear in quotes using the general form of a target expression, $("anything goes").

C++ Issues

APC, the "language" in which you write your probes, is an extension of ANSI C, and is preprocessed into C before being compiled into a UAL. You cannot use C++ syntax directly.

Furthermore, you cannot generally compile C++ object files with your C++ compiler and link them into your UAL with apc. This is because the C++ would have references into your target program and to the C++ runtime which cannot be resolved within the UAL. If you need to reference C++ in your probes, contact OC Systems.

It is possible to create C++ objects and call the constructors in a target program from a probe, but it requires a thorough understanding of how the C++ object would be represented in C. This is very compiler-specific and is recommended only for advanced users.

But you can still get a lot of information about your C++ methods, its parameters and local data. And you can invoke other methods which are in the same class as the probed method, or for which you have somehow saved the "this" pointer.

C++ Class Data (this)

Each non-static data item in a class is really a component of the object pointed to by the "this pointer", which is passed as the first parameter to each method in the class. This has the following implications:

  • All dynamic class data must be preceded by $this->, which is defined even if no executable is provided to apc.
  • Static class data must be referenced directly, and will not be shown as part of the data logged with $* or $this.
  • Entity names which contain a class modifier (e.g., method names and static class data) must use the general (quoted) form of target expression, e.g.,
    $("CarClass::NDoors").

C++ Class Method Calls

In order to call a method within a class, you must supply a C++ object of that class ($this) as the implicit first parameter. This generally limits you to calling a method in the same class, though you can also save a pointer to (or copy of) an object in one probe and use it in another. A common use of a method call is to invoke a debug-output method as illustrated in the example below.

probe thread 
{
  probe "ListClass::AddToList(int)"
  {
    typeof($this) ThisPtr = $this;
    on_entry
      $("ListClass::DebugDump(char*)") (ThisPtr, "Before adding int:");
    on_exit
      $("ListClass::DebugDump(char*)") (ThisPtr, "After adding int:");
  }
}

Example 3-4. Calling a C method from a probe

Ada Issues

Ada data names

There are several points to remember when accessing data items defined in the probed Ada program:

  • Selected (dotted) names must use the general (quoted) form of target expression.
  • Simple names and subcomponent names must be in lower case.

PowerAda unit names

PowerAda units cannot be referenced by their file name (because the file name information is not available). Instead, use the -unit context description to name the unit. The context description would look like:

  • "-unit lib/my_package" to reference the spec of "my_package"
  • "-unit sec/my_package" to reference the body of "my_package"


GNAT Ada Out mode parameters

For PowerAda on Linux and AIX, Out and In Out mode parameters may be referenced by name just as In parameters. However, for GNAT it's more cmoplicated.

There are three cases that must be recognized when accessing Out mode parameters from a GNAT-compiled Ada procedure: single scalar Out parameters, multiple scalar Out parameters, and Out parameters that are passed by reference.

Records and arrays are generally passed by reference. This means that the address of the argument is passed as an In parameter; and the contents of that address are updated within the called procedure, but the address itself is not changed and hence doesn't need to be returned. In this case, you simply reference the Out parameter by name in the on_exit part to display it or change it.

For example, for:

procedure p(x : out record_t);

simply use:

probe "pkg.p"
{
on_exit
log("x = ", *x_addr);
}

The GNAT implementation of scalar (pass by-value) Ada Out parameters is such that logging $return ($0) will log all out parameters as fields of a structure, except that if there is just one Out parameter, then it's the return value itself rather than a field in a structure.

So, for procedures with a single [in] out parameter, such as

 procedure p(x : in fred_t; y : out fred_t);

Use:

  ...
  on_exit
    log("y = ", $return)

For procedures with more than one [in] out parameter, such as:

  procedure q(x : in fred_t; y : out fred_t;quit : in out boolean);

Use:

  ...
  on_exit
    log("y = ", $return.y)

Also note that:

  • Out parameters cannot be accessed by position other than $0 ($return); and
  • you must skip any OUT-only parameters when determining the position of an IN parameter.

So in procedure q above, you would have:

...
on_entry
  log("quit = ", $2);

It's generally easier to just reference parameters by name as long as debug information is available (i.e., files are compiled with -g).

Ada protected objects

For PowerAda-compiled programs, functions and procedures within protected objects are not generally accessible, although in simple cases a protected procedure may map to a normal procedure of the same name.

In a GNAT-compiled Ada program, A protected subprogram is implemented by a function which must be referenced by its mangled name. The protected data is implemented as a record passed by reference as the first parameter ($1). Its name is $object. Its type is an internal GNAT type. While rather awkward, it is possible to probe the protected subprogram and access the data, as illustrated in Probing a Protected Procedure

Given the protected object Prot_Type.Result:

package Prot_Type is
  protected Result is
    procedure Set (I : in Integer);
    function Get return Integer;
  private
    The_Int : Integer := 100;
  end Result;
end Prot_Type;

Here is a probe to check that Set correctly assigned the input parameter to The_Int:

probe "prot_type__resultPT__setP" 
{
  // grab the input value
  int i = $i;

  on_exit 
 {
    // check that the assignment worked.
    // the protected object is "_object"
    if ($_object.the_int != i) 
    {
      printf ("Prot_Type.Result.Set didn't work.\n");
    }
  }
}

Example 3-5. Probing a Protected Procedure

Nesting of Probes

Probes may be lexically nested. The visibility and activation rules follow from the nesting in a manner that is similar to structured languages like Ada, Pascal and Algol. Most often, only the simplest of nesting is used, but the ability to nest probes can be a very powerful feature.

Probes are nested to control the activation of the probes and to control the "upscope" visibility of the data items declared in the probes. Each probe directive has a declarative region that appears directly after the opening brace. This is similar to C and C . Both probe directives and regular C declarations may be placed in this declarative region and referenced from within the body of the probe, again similar to C.

Unlike C visibility rules, but similar to structured languages such as Pascal and Ada, the probe directive may be nested within other directives, and the nesting of data items may also occur.

More interesting and powerful than the scoping of data within probes is the concept that an outer probe provides an umbrella for a nested probe. That is, the inner probe is executed only if the enclosing probe is active. This means that if one probe function is within another, the inner one is executed only if the function probed by the outer one is being executed.

Note that nesting of probes may have surprising results when the functions being probed are invoked recursively--see Nested Probes and Recursive Calls.

The following rather lengthy example illustrates this umbrella concept as well as the nesting of data items.

Example of Probe Nesting and Visibility

Suppose we have an application which is a database server. Whenever a client connects to the server, a new thread of execution is created to handle that client's queries. We want to keep track of all deletion of records in the database, and who's doing them. However, the DeleteRecord() operation is called both directly by DoQuery() and by ChangeRecord(), and we don't care about changes. The client is prompted for identification and his reply is received by the local function RegisterClient() and we can record the client's identification from there. Nested probes and Visibility Rules is a probe that does it.

A static variable named TotalDeletesFromAllThreads is declared at the outermost level, outside of both probes. This variable (in standard C fashion) is visible to all code in the file. The probe program will initialize this variable to zero at program start and will print its contents to standard output upon program termination.

The probe thread directive contains two nested probes. For each new thread that is created in the application, this probe thread is activated. Each newly created thread will get its own probe thread. For example, the data items declared inside the probe thread are allocated on a per thread basis, so each thread has its own copy of TotalDeletesInThisThread. In this way, it's easy for the user to declare per-thread data so that multiple data access is not a problem.

The body of the probe thread initializes its local data on entry to the thread, and it prints out the values upon the exit of the thread.

Declared inside the probe thread is a nested probe on DoQuery(). This probe will be executed every time that DoQuery() is called. The body of the probe on DoQuery() sets some variable values and also declares a nested probe for the method DeleteRecord().

The key point is: this nesting of one probe inside another ensures that we only probe DeleteRecord() when it is called from (anywhere) inside of DoQuery(); and it has nothing to do with whether the actual function DeleteRecord() was really declared inside of, outside of, or far away from, the actual function DoQuery().

The probe on DeleteRecord() is only active when the probe on DoQuery() is active. We call this "opening an umbrella". The probe on DeleteRecord() will only be invoked when DoQuery() has been entered but has not yet returned. This example makes use of this fact; it collects (and prints out) the number of calls to DeleteRecord() for each invocation of DoQuery(), and it also tracks the total for all such calls.

Calls to DeleteRecord() that are not under the umbrella of DoQuery() (e.g., do not have DoQuery() somewhere in their call chain) do not activate the probe on DeleteRecord(). So DeleteRecord() could be called hundreds of times from other parts of the program and this probe would never be executed. If you desired to catch all calls to DeleteRecord() (not just those under the umbrella of DoQuery()), then the probe on DeleteRecord() should be moved out to the declarative region of the probe thread, so it's outside of the probe on DoQuery().

For more detailed rules concerning nesting and visibility, see Probe Nesting and Visibility Rules.

#include <stdio.h>

static int TotalDeletesFromAllThreads;

probe program
{
  on_entry
    TotalDeletesFromAllThreads = 0;
  on_exit
    printf("Total number of Deletes was %d\n",
      TotalDeletesFromAllThreads );
}
probe thread
{
  int TotalDeletesInThisThread = 0;
  char *ClientName = NULL;

  probe "client.C":"RegisterClient()"
  {  // first parameter is user's name
    on_entry
      ClientName = ap_StrDup($1);
  }
  probe "QueryHandlerClass::DoQuery()"
  {
    int DeletesInOneQuery = 0;
    probe "DeleteRecord()"
    {
      on_entry
      {
        DeletesInOneQuery  ;
        TotalDeletesInThisThread  ;
        TotalDeletesFromAllThreads  ;
      }
    }
    on_exit // from DoQuery
      if (DeletesInOneQuery > 0)
        printf("Query deleted %d records.\n",
          DeletesInOneQuery );
  }
  on_exit  // from thread
    if (ClientName) // print results for this client
      printf( "Client %s did %d deletes.\n",
            ClientName, TotalDeletesInThisThread);
}

Example 3-6. Nested probes and data

Logging and Formatting Data

In the example above, we see that we can print out the Aprobe collected data immediately using the printf() function. We could also have done many other things with the data, like: use system calls to open files and write the data directly to them, open a pipe and write the data to another program, write the data to standard error or anything else we choose, since we are writing our probes in C.

In many cases the most useful thing to do with Aprobe collected data is to log it. This means that the raw data is captured and recorded in a low-overhead manner for formatting, filtering and output at a later time.

Some power users may want to do such things as log to a different device instead of a disk file (perhaps a network attached device) or format the data without having the original UALs around. These advanced logging capabilities are not covered in this User's Guide, but you are encouraged to contact OC Systems for documentation and assistance in using the underlying general-purpose logging support.

Target Expressions (the references starting with $) are described above. Logging with automatic and user-supplied formats is described in following sections.

The syntax of log statements are fully defined in Probe Syntax, but the following are examples of log statements:

log();    // a no-op, but legal

log(x);  // log APC variable x with an automatically
          //generated format

log($*); // log all parameters to probed function with an
          // automatically generated format.

log ("On entry: ", $x) with xformat;
         // log string and target variable x with
          //user-defined format

Example 3-7. Log statements

Automatic Data Formatting

The log statement, in its simplest form, uses the Aprobe runtime to write user-selected data out to a disk-resident file in a minimally invasive manner and to invoke the correct (automatically generated) formatting routine for that logged data item when apformat is run.

probe thread
{
  probe "fred"
  {
    on_entry
      log( "My variable is ", $my_variable );
  }
}

Example 3-8. Log with automatic format

The Aprobe preprocessor will produce all the code necessary. For example the probe in the example below will result in the string "My variable is" and the value of my_variable (from the application name space) being logged to the Aprobe data file. Later, when apformat is run on this data file, the string "My variable is" will be followed by the actual value of my_variable, and that will be printed out. The value of my_variable will be automatically formatted according to its type in the application name space. For example, if the type of my_variable is float, then the data will be formatted as a floating-point value. If it is a structure, then each field of the structure will be named and correctly formatted.

So in its simplest form, Aprobe automatically figures out how many bytes to log and it figures out which format to use for those data when apformat is run.

User-Supplied Formatting

The user may desire to supply a formatting routine for the data, in place of the automatically generated one. This is quite common. There are a number of reasons to do this.

  1. The automatically generated format may be too verbose. It contains the field names for each field of the output struct, for example and while this makes it simple to read the output, it can be verbose when you are logging this millions of times.
  2. You may need a format different from the one that is automatically generated. For example, it is common to process the output from apformat using other tools. If one precedes some logged data with a unique string at format time before printing it out, utilities such as Unix awk and grep can easily be used to search for that unique string.
  3. You may desire to use the formatting capability of Aprobe to do data analysis. The Aprobe log statement acts like a deferred function call. At run time, the specified data is safely written to the APD file. At format time, the correct format routines are called for each logged data item in the exact same order that they were logged, with the logged data passed as parameters to the format routines.

It is common to have these format routines do processing on the data, and not actually print anything out at all. Perhaps just the summary data condensed from all the millions of logged data items are printed out by a probe format on_exit action.

Each formatting subprogram is a normal C function whose parameter profile corresponds to the actual parameters on the corresponding log statement. When apformat is run, the correct formatting subprogram is called, once for each log entry, in the same order that the corresponding log statements were executed.

To specify your own format, you write a function that has the same prototype as the arguments to the log directive except that each argument to the format function is formally declared as a pointer to the type.

Format Routines Have Pointer Parameters

This is very important: The format routines are called with a pointer to the logged data, not the logged data itself. The apc preprocessor and C compiler check to make sure that the parameters for the log and the user-specified format function are compatible.

Also, note that there is nothing special about any user-specified format routine; it is a normal C function.

 float  f;
 int    i;
 struct wilbur {
   int a;
   ...
 } w;
 
 void my_float_format(float *arg) {
   printf("%f\n", *arg);
 }
 void my_int_format(int *arg) {
   printf("That's quite a saving: %d\n", *arg);
 }
 
 void my_wilbur_format(struct wilbur *arg) {
      ... print the various fields of wilbur ...
 }
 
 void my_format(float *p1, int *p2) { ... }
 
 probe thread
 {
   probe "fred"
   {
     on_entry
     {
       log(f) with my_float_format;
       log(i) with my_int_format;
       log(w) with my_wilbur_format;
       log(f,i) with my_format;
     }
   }
 }

Example 3-9. User-supplied format routines

Automatic Formatting within a User-Defined Format

You can use automatic formatting within a user-defined format routine using the log directive. This is because "immediate formatting" is always in effect at format-time, when the format routine is invoked. So a "log" statement within a format routine is just a direct call to that log statement's format routine. This can be very useful if you want to display some components of a structure in a special way, but use the default for others, for example:

 static void MyFormat(typeof($ThreadHandler) *T)
 {
   if (T->ThreadIsValid)
     log("MyFormat => ", *T);
   else
     printf("Invalid thread\n");
 }
 
 probe thread
 {
    probe "ThreadHandler::CheckThread(class ThreadHandler*)"
    {
       on_entry
          log(*$1) with MyFormat;
    }
 }

Example 3-10. Using log in a format routine.

Logging Multiple Objects

As shown above, multiple values may be specified as log parameters, for example:

 log (10, $foo) with MyFormat;

This log statement will store the value 10 and the value of the variable foo (where foo may be a C structure, in which case the Aprobe runtime will log the entire structure). Each log_parameter will be evaluated exactly once and will be logged in a single operation. Later, at format time, the subprogram MyFormat will be called with these two values. Note that the Aprobe compiler will check that the format subprogram MyFormat is defined and accepts two parameters of the correct types.

Note that $* (representing all parameters to a probed function) is potentially expanded into multiple log parameters, so it should be used only with automatic formatting.

Logging Pointers

In understanding the log operation it is important to understand that the data described by the log parameters must be copied from the program memory to the APD file, because the program will be finished and that memory unavailable when the data is formatted. So the apc compiler must know exactly how many bytes to copy to the APD file, and pointers are NOT followed when determining this. If you do "log(StructPtr);" you'll get a numeric address; if you do "log(*StructPtr)" you'll get all the fields of that structure, but if one of these is itself a pointer, you will get the numeric address value. This is very similar to what a symbolic debugger will give when you query the value of such values.

Logging Arrays

For composite items other than arrays, one only needs to specify the actual item, since the apc compiler can determine from the variable type the number of bytes to log. For arrays, however, it is necessary to specify which elements are to be logged.

The preprocessor implements an easy-to-use syntax for specifying array lengths using " .." (two adjacent dots, preceded by a blank) between the first and last indices of the array to be logged, similar to the range operator in the Ada programming language, as the following example illustrates:

 static int Arr[10];
 void my_format (int *p, int *n)
 {
   int i;
   for (i=0; i<*n; i  )
     printf("%d,\n",p[i]);
 }
 probe thread
 {
   probe "foo"
 {
     on_entry
       // log the first 6 elements of Arr with my_format
       log(Arr[0 ..5], 6) with my_format;
   }
 }

Example 3-11. Logging an array

At format time, my_format will be called with "p" pointing to an array of 6 integers. Note that this array logging can also be used with the automatic format generation. For example:

     log(Arr[0..5]);

Logging C Strings

A C null-terminated string is a special case of array that is addressed by the macro ap_StringValue. Use it like a function to log any null-terminated string value:

 probe thread {
   probe "main"
   {
     on_entry
       // log the executable name:
       log("Entering main() for: ",
         ap_StringValue(*(char **)$2));
   }
 }

Example 3-12. Logging a string

Logging Ada Strings

Ada unconstrained strings and other arrays are recongized by Aprobe, so generally they need not be treated differently than C strings, except that the lower bound is generally 1 rather than 0. In fact, because of the additional type information available from Ada, the entirety of such strings and arrays can be logged without specifying their bounds.

Given the following procedure:

 procedure Print(S : in String) is
 begin
    Ada.Text_IO.Put_Line(S);
 end Print;
 

You can simply write:

 probe thread
 {
   probe "print"
   {
     on_entry
       log("print: ", $S);
   }
 }

Example 3-13. Logging Ada Dynamic Strings

The Aprobe API

The apc compiler translates all the probe directives into ANSI C source code containing calls to the Aprobe Application Programming Interface (API). The interface to this API is defined in the header file $APROBE/include/aprobe.h, which is implicitly #included into each APC file, and documented in Aprobe API Reference.

Because of the powerful directives, you will need to code very few references to this API directly. However, in some examples you may see references to data and functions that start with "ap_". These are part of the Aprobe API. If you need some information about the state of the current program within a probe, chances are there's a way to get that information through a function or macro in aprobe.h.

Below are some of the simpler and more useful operations defined there. For the full specification, and other supporting functions, refer to the Aprobe API Reference or just use your editor to look at the aprobe.h file directly--it is documented with the intent of being a stand-alone reference.

Stub Support

The macro ap_StubRoutine is provided to make it easy to disable, or "stub," a function so the body of the function is not executed. Control is transferred directly from the on_entry action where the ap_StubRoutine appears directly to the on_exit actions (if any) where a return value may be specified.

The macro takes no parameters (and is not a function so needs no parentheses). For example, to stub out a function and have it return 0:

probe "foo"
{
   on_entry ap_StubRoutine;
   on_exit $return = 0;
}

Example 3-14. ap_StubRoutine

Note, however, that the on_exit portion of the probe must provide suitable values for results expected from the stubbed routine. You cannot set the return value on_entry, even if you know the calling conventions, since the return register is overwritten as part of ap_StubRoutine.

It should also be noted that on Solaris a structure returned by value is written to space on the stack allocated by the caller. However, if the caller is discarding the returned value by calling the function as a procedure, no space is allocated. In this case, a probe which may normally attempt to change the return value should not do so, as it will likely corrupt memory. In order to allow users to handle this problem, the following macro is provided:

#define ap_StructValueReturnExpected private

This would be used as a boolean expression in an on_exit part as follows:

probe "UpdateCoordinates()"
{
  on_entry
    ap_StubRoutine;
  on_exit
    if (ap_StructValueReturnExpected)
      $return.x = $return.y = $return.z = 0;
}

In fact, the routine need not even be stubbed for this to be a problem, so this should be done in general with functions that may return a struct by value.

Stack Trace Support

The traceback functions and macros provide an easy mechanism for obtaining a stack trace--a list showing the current routine, the routine which called that, its caller, and so on up the stack. The simplest way to do this is to use these macros:

ap_LogTraceback(depth)

Log the current stack trace back depth levels for later formatting.
ap_PrintTraceback(depth)

Print the current stack trace immediately.

ap_LogTraceback and ap_PrintTraceback take only one parameter: the depth, which is the number of levels back from the current location to trace. (The depth argument must be an integer constant since it is used to declare a C array of that size.) These macros can only be used within a probe action directive (on_entry, on_exit, etc).

probe "Database::Update"
{
  on_entry
  {
    if ($1 == 0)
    {
      log("*** Update called with null pointer!\n");
      ap_LogTraceback(10);
    }
  }
}

Example 3-15. ap_LogTraceback

Exception Support

The C++ language and Ada languages define exceptions, which transfer control from a procedure or function in a program (a throw, or raise) to some point up the call chain where it is "caught" or "handled" or perhaps terminating the program. Aprobe provides some subprograms to log and/or print exceptions as they occur.

Exception logging and printing are enabled on entry to a thread by registering a handler using ap_RegisterExceptionHandler(). This is simplified by three parameterless macros:

ap_LogExceptionsInThread

Log all exceptions which occur in the current thread.
ap_PrintExceptionsInThread

Print (at run time) all exceptions which occur in the current thread.
ap_LogAndPrintLogExceptionsInThread

Both log and print all exceptions which occur in the current thread.

One of these may be placed before the closing brace of a thread probe:

probe thread
{
  /* your function probes */
  probe "foo"
  {
  ...
  }
  ap_LogAndPrintExceptionsInThread;
}

Example 3-16. ap_LogAndPrintExceptionsInThread

Also, when an on_exit action is called, it is possible that an exception is being propagated. You can use the implicit parameter ap_ProbeActionReason to check this:

if  (ap_ProbeActionReason == ap_CppExceptionPropagated ||
   ap_ProbeActionReason == ap_AdaExceptionPropagated)
{  ... do exception-related actions ... }
else
{  ... do normal actions ... }

Example 3-17. Checking ap_ProbeActionReason for exceptions

If you are using the IBM xlC compiler on AIX, you can prevent an exception from being propagated past a function by using the ap_SuppressException macro in an on_exit probe on that function.

It can be used only on_exit, when ap_ProbeActionReason == ap_CppExceptionPropagated. For example:

probe "fred"
{
  on_exit
    if (ap_ProbeActionReason == ap_CppExceptionPropagated)
      ap_SuppressException;
}

Example 3-18. Using ap_SuppressException

In addition to getting information about exceptions, Aprobe supports raising exceptions at the point of a probe as well. This is currently supported only for Ada programs, using the appropriate macro for your compiler:

ap_BooleanT ap_RaiseGnatException(ap_NameT);
void ap_RaisePowerAdaException(ap_NameT);

This simply raises the named exception within an action; for example:

probe "List_Ops.Remove_Head"
{
  on_entry 
  {
    ap_RaiseGnatException("List_Ops.List_Is_Empty");
  }
}

Time Support

The Aprobe API defines a target-independent time type, ap_TimeT, and a number of functions to operate on this. It is represented as seconds and nanoseconds, though the actual time values may not be accurate to the nanosecond. See aprobe.h for a full definition, but the following are generally sufficient for timing applications:

ap_TimeT *ap_GetTime (ap_TimeT *Time);

Put the current time value into the Time buffer provided by the caller, and return the address of that buffer.

ap_TimeT ap_GetCurrentTime();

Return the current time.

ap_TimeT ap_AddTime (ap_TimeT T1, ap_TimeT T2);

Add two times: Result = T1 T2.

ap_TimeT ap_SubTime (ap_TimeT T1, ap_TimeT T2);

Subtract one time from another: Result = T1 - T2.

float ap_TimeToFloat (ap_TimeT Time);

convert Time to a float value of the form seconds.nanoseconds.

char *ap_FormatTime(char *Buffer, ap_TimeT *Time);

return a string representing the time. See also ap_GetTimeImage().

The automatic log formatting recognizes the ap_TimeT value and prints it using ap_FormatTime(). Using these functions, one can measure and display how long a subprogram takes to execute:

probe "foo"
{
  ap_TimeT EntryTime;
  on_entry
    log("Entered foo at ",
      (EntryTime = ap_GetCurrentTime()));
  on_exit
    log("foo took ",
       ap_SubTime(ap_GetCurrentTime(), EntryTime));
}

Example 3-19. ap_Time operations

Periodic Action Support

Aprobe contains built-in support to invoke a user-specified function every so many seconds while your program is running. This has a number of uses, including watching a variable, dumping a buffer, or updating a display. This support is provided by the function ap_DoPeriodically.

To use it in an APC file, define a regular C function that takes a single parameter of type void *, as described by the type ap_PeriodicActionT:

typedef void (*ap_PeriodicActionT) (void *);

Then, in the on_entry action of a probe program register this function to be called periodically, passing the function name, the number of seconds, and the data item to be passed to your periodic function.

ap_BooleanT ap_DoPeriodically(
    ap_PeriodicActionT   UserAction,
    int           IntervalInSeconds, 
   void            *UserData);

This returns TRUE if the action was registered successfully, FALSE otherwise.

The $APROBE/examples/learn/visualize_data] example demonstrates the use of this, in conjunction with the quick_gui.ual library.

Tracing Support

Aprobe includes some macros for easily recording execution of entry, exit and lines: ap_LogSubprogramEntry, ap_LogSubprogramExit, and ap_LogLine. For more information, see Tracing Support and see the example below.

probe thread
{
  probe extern:"main()"
  {
    on_entry
    {
      ap_LogSubprogramEntry;
    }
    on_line (3)
    {
      ap_LogLine;
    }
    on_exit
    {
      ap_LogSubprogramExit;
    }
  }
}

The output of executing the above probe would be:

[Enter : extern:"main()"
Called from ==> extern:"_start()"   0x00b8
 at 08:46:15.635003390
  @:3 at 08:46:15.635049010
]Leave : extern:"main()"
 at 08:46:15.635067822

Note that the "[Enter" and "Called from" lines are gneerated automatically by ap_LogSubprogramEntry. ap_LogLine generated the "@:3" line, and ap_LogSubprogramExit generated the "]Leave" line.

As with all probes you can provide alternate formats for these, for example, to generate tablular output. Contact OC Systems for assistance.

Debugging Your Probes

It occasionally happens that you write an APC file and the apc preprocessor or C compiler fails to compile it. Or, you may successfully create a UAL file but when you run aprobe on your executable with that UAL file, the program crashes and dumps core. The purpose of this section is to provide some guidance in finding and fixing the problem.

Compile Time Errors

The apc preprocessor may report errors if a name is not defined, or there is a syntax error. These are similar in format to those printed by the C compiler, though the messages generally don't give much help as to the cause of the problem. If you are unsure as to whether a message is from the apc preprocessor or the C compiler, you should invoke apc with the -v (verbose) option, which will show each command as it is run.

Problems resolving target expressions

A common error from the apc preprocessor is "name not resolved". This means that a target expression identifier could not be resolved using the executable or shared library provided with the -x option on the apc command line. You should check the following.

  • The correct executable or library was given to the apc compiler (see apc);
  • The executable is not stripped (that is, has not had symbols removed using the strip command).. You can use the file command to determine if the file is stripped as follows:

file a.out

  • The executable contains debug information; in particular, that the file containing the name you want was compiled with -g.
  • Tf the identifier is local to a file, it may require "-file" context information in order to resolve it (see Target Expressions).
  • If the identifier is local to a function, it is visible only from the on_line actions in that function. Local data is not available from on_entry or on_exit.

A good general test for whether a target expression can be determined by the apc compiler is whether it can be resolved by the source-level debugger. If dbx can give you a value for a data item at a certain point, Aprobe should be able to as well.

The apcgen program will identify what symbols are available in an executable, and what symbols have debug information:

apcgen -vL foo.exe
apcgen -dvL foo.exe

The first lists all functions, whether or not they have debug information. Adding the -d flag shows only those functions with enough debug information to generate a probe for. The names listed by apcgen are those that should be used in a probe statement. The -v flag adds source file and line information (if available) to help distinguish between similarly-named functions.

C compiler errors

If an error message is generated from the C compiler after successfully preprocessing the APC, the line numbers still refer to the APC file. If it is unclear which APC statement is generating the error, you may wish to compile the output of the apc compiler directly as follows:

  1. Specify the -C or -g option so that the generated intermediate C is saved:
    apc -C fred.apc -x fred.exe
  2. Edit the intermediate C file, fred.apc_c.c, to remove or comment out the #line directives, so error messages refer to the C file itself, and put the result in a different filename. You can do this at the command-line as follows:
    $ grep -v "^#" fred.apc_c.c > fred_apc.c
  3. Compile the resulting C file:
    apc -g fred_apc.c

The error messages will now refer to the generated C.

C compiler warnings

It is important to take note of warnings generated by the C compiler. Nearly always these reflect legitimate problems with type mismatches, especially between logged data and the corresponding format routine parameters, or between the type of a target expression and the type of an APC expression to which it is assigned or passed.

A common source of mismatch between log arguments and user-supplied format parameters is that scalar values in the log statement must have pointer-to-scalar parameters in the format statement.

Run Time Errors

The more difficult errors to track down are those which simply cause a core dump when you execute your program with aprobe.

Adding Print Statements

A very valid way of debugging your probes is to insert print statements in your probes, recompile them with apc, and rerun them with aprobe. You can put a printf() at the start of every probe action, display the values of pointers, and subexpressions, and target expressions--even call assert(). These messages will print as the probes are executed, and can quickly tell you if a value is not what you thought it would be.

Using gdb or dbx

If you can't track down the problem with print statements in your probe, you're probably going to need the same debugger you use to debug your application. We'll assume it's gdb, though it may be wrapped in a graphical interface that hides some of what we discuss here. All descriptions here apply to dbx as well..

Generating debug information for probes

The apc compiler is like the C compiler (which it invokes): it saves debug information in each UAL only if you specify "-g". You should use the -g flag always when developing probes, so you're ready to debug your resulting UAL file when necessary. Using -g also saves the C source code generated by the apc compiler, although the debugger will still show you the APC source as you step through your probe.

Using gdb with core files

If there was a core file produced when you ran aprobe (but the program executed to completion without probes), simply invoke gdb on your target executable and this core file:

gdb foo.exe core

When you get a prompt, enter the where command. You have a limited ability to examine values, but it's a quick way to get an idea where the problem is. For example, it may show you that you called printf() with null pointer.

Using gdb with aprobe

If you need to interactively debug the program, you can pause the application in your probe and attach to it. For example, you can write a probe (or edit into your probe) something like this:

#include <unistd.h>
probe thread {
  probe extern:"main()" {
    on_entry {
      printf("Attach debugger to %s, process %d\n"
          "Waiting...", *(char **)$2, getpid());
      pause();
    }
  }
}

Example 3-20. pause_in_main.apc

If you're debugging a C program compiled with xlC on AIX, and wish to debug code executed before main is reached, use the following entry point instead of "main()" in the probe above: "__C_runtime_startup()".

If you're using g , all the C startup is done after main() is entered.) You can put this in a separate probe that you have sitting around, and then just add that to your aprobe command, e.g., in one window:

$ apc pause_main.apc
$ aprobe -u my_probe.ual -u pause_main.ual my_program
Attach debugger to my_program, process 102328
Waiting...

On Solaris you can specify -deb on the aprobe command line which will pause the debugger before any probes or any of the program is executed.

In any case, to attach with gdb you can do:

gdb foo.exe 12345

At the gdb prompt you can set breakpoints on any functions in your program or your probes. The program has not yet started.

Setting breakpoints

The C source file that is saved when you specify apc -C is created in the local directory using the APC file name with _c.c appended, (e.g., foo.apc_c.c). In this file are functions with names like OnEntry_0001_L0027 and OnExit_0001_L0027. Each of these corresponds to a probe action; these names, for example, correspond to the entry and exit actions of a probe which starts at line 27 of the APC file. You can set breakpoints on these functions and step through them to follow the execution of your probes.

Using a debugger on a stand-alone probed executable.

if attaching to the process seems clumsy, or you need to look at the program during some early part of the Aprobe runtime's execution, you can use the libdal.so shared library and an APO file to create a stand-alone probed executable which can be run and debugged directly. See Loading Probes Without Aprobe for a description of how to do this.

NOTE: this mechanism works only for Solaris versions 2.6 or newer. In particular, it does not work for Solaris 2.5.1, because starting the program under dbx prevents the aprobe dynamic patching from taking place.

When using the debugger on a probed executable, or apformat, it is important to understand that the probes and format routines are not available when the program starts, nor after it has terminated. See teractively debugging apformat below for more details about this.

Format Time Errors

apformat is similar to aprobe in that it reads from both the executable and the UAL files and it executes code from the UALs. apformat won't run the executable and the UAL code it executes are just the format routines, but it's very possible that crashes and core-dumps may happen when you run apformat.

Debugging apformat with a core file

To use the core file, you can debug apformat just as you did your target executable:

gdb $APROBE/bin/apformat core
where

This will show you where the problem is. Nearly always this is one of your format routines trying to print a NULL or invalid value that was logged at runtime. You can make this kind of problem happen at run time, and so determine which log statement was responsible, by rerunning aprobe with the -if option, which directly invokes each format routine with the logged data rather than writing to a file.

Interactively debugging apformat

When using the debugger on apformat, it is important to understand that the format routines are not available when the program starts, nor after it has terminated. You can query symbols defined in them (functions or variables) after a certain point. Here is approximately what apformat does on startup. This is very similar to what aprobe does:

  1. Read the command-line parameters and locates all the files needed to execute. These include the Aprobe runtime, the specified UAL file(s), and the target executable program.
  2. Dynamically load the runtime with a call to dlopen();
  3. Load the UAL file(s), also by calling dlopen().
  4. Locate and open the APD files containing the logged data.
  5. After these steps are performed, apformat runs the on_entry actions of the format probes.

You must wait until step (3) is completed before the functions and variables in your format routines (in the case of apformat) or probe actions (at aprobe runtime) are available.

We've not figured out a way to break on dlopen() in gdb, so you're on your own there. However, the following examples for dbx should be sufficient if you're a gdb guru. The following should work in dbx on Sun Workshop 4.0 or newer:

  1. Invoke dbx:
    $ dbx $APROBE/bin/apformat
  2. Set apformat command-line options:
    (dbx) runargs foo
  3. Set a one-time breakpoint when shared libraries are loaded, then set a breakpoint right before executing the format routines:
    (dbx) when dlopen -temp {
    stop; stop in ap_RunFormatEntryProbes ; }
  4. Start execution of apformat:
    (dbx) run
  5. The first dlopen breakpoint is for the Aprobe runtime.
    Continue on to the break at ap_RunFormatEntryProbes():
    (dbx) cont
  6. At the ap_RunFormatEntryProbes breakpoint, all the UALs are loaded, and you can set breakpoints on your format routines:
    (dbx) stop in foo_format

Similarly, the following should work in dbx on AIX.. The prompts are included below to make clear what are shell commands and what are dbx ones. It is assumed you are using the dbx debugger that accompanies the xlC compiler:

  1. Invoke dbx:
    $ dbx $APROBE/bin/apformat
  2. Set a breakpoint when shared libraries are loaded by calls to dlopen():
    (dbx) stop in dlopen()
  3. Start execution of apformat:
    (dbx) run foo
  4. The first 2 dlopen() breakpoints are for the Aprobe runtime, and the third is for the first user UAL that is to be loaded. You can see this by examining the first parameter to each dlopen() call:

[1] stopped in dlopen at 0xd018dd54
(dbx) ($r3)/s
20001c88: "/usr/local/aprobe/lib/system_ual.a(system_ual.o)"
(dbx) cont
[1] stopped in dlopen at 0xd018dd54
(dbx) cont
[1] stopped in dlopen at 0xd018dd54

  1. When you hit dlopen() for the third time, you can set a breakpoint at ap_RunFormatEntryProbes(), delete the breakpoint that was on dlopen(), and continue:
    (dbx) stop in ap_RunFormatEntryProbes
    (dbx) delete 1
    (dbx) cont
  2. At the ap_RunFormatEntryProbes breakpoint, all the UALs are loaded, and you can set breakpoints on your format routines:
    (dbx) stop in foo_format

Next Previous Top Contents Index

Copyright 2006-2017 OC Systems, Inc.