RootCause/Aprobe FAQ

11.1 I applied a Trace on function (method) in the RootCause GUI, but I don't see it being called in the output. Why?

Here are some possibilities:

• The function was called so often that it was load shed, and calls stopped being recorded. Click on the LOAD_SHED node at the end of your Trace Display, choose Show Associated Table, and look for your function there. Using the option-menu in the first column can designate the function as Do Not Shed for subsequent runs.
• The function was called, but it's not shown in the data file you're viewing. Use Add Data Files to Display in the File menu to add earlier files. If you still don't find it, then the data containing the last call may have been overwritten (i.e., the "trace buffer wrapped around"). You can save all data files containing the trace of a function by adding a SNAPSHOT probe ON_ENTRY to the function in the Trace Setup dialog.
• There are multiple instances of the method in different classes, and you chose the wrong one. Use Find in Trace Setup and set traces on others that occur.

The following possibilities apply only to native (C/C++) functions:

• The function that's really being traced is in a different module. For example open() in libthread.so instead of open() in libc.so (see Q11.3 ). Use Find in trace setup and set traces in all modules where your function appears.
• You did "Trace All In" which generates a wildcard, but the function was one of those that's not traced as part of a wildcard because it requires an expensive "trap" patch. Return to Trace Setup and force an explicit trace on this function by adding a "probe" on entry.
• The function was optimized and so "inlined" at the point of call. If there really is no call, the function can't be traced.
• The function cannot be traced. There are a few functions that because of the way they're coded simply cannot be probed. To test for this, go to the command-line and type:

  apinfo -sa -x your_application.exe | grep "your_missing_function"

• or the Windows equivalent. If you see your missing function in the output, it cannot be instrumented. Contact OC Systems to find out why.

11.2 I add a library as a dynamic module and trace the init function, but the trace doesn't show up. Why?

When you add a module as a dynamic dll, this forces it to be preloaded (loaded before program start rather than at the point of the dlopen() / LoadLibrary() ). This means that the _init() function is called before _start of your main application, which is before probes have been applied.

11.3 Solaris: I'm trying to trace file open calls, so I trace "open()" in "libc.so" , but I get nothing. Why?

On Solaris, there is an open() in libc.so and one in libthread.so . They both call something called libc_open() in libc.so , so that's the one you should trace. In RootCause 205 this made more accessible using the shadow header file, so will show up as residing in source file libc.so.h . There's also a libc_close() .

11.4 Solaris: I Add Dynamic Module of mylib.so, then specify some traces in mylib.so. But when I run the program, those traces don't appear. Why?

You may be loading a different instance of the library at runtime than you specified to Add Dynamic Module. This may be the case if LD_LIBRARY_PATH (or equivalent) is set. Make sure that the full path to mylib.so you've added to your workspace is the same as the one that will be loaded at runtime. On Windows, change ".so" to ".dll" and LD_LIBRARY_PATH to PATH and this still applies.

11.5 I did Custom..., and saved my probes to an APC file, but those probes don't show up in my trace. Why?

Make sure the "Add to Custom APC Files" checkbox is checked. If you've already got an APC file, make sure the Append checkbox is checked as well. Also, see Q11.1 .

11.6 Windows: Why doesn't the DOS dir command show my workspace, pi_demo.aws?

Because workspaces are marked as System Folders, so we can associate a special icon with them. Of course this doesn't help when you're in dos mode. Use dir /as to see system folders.

11.7 Windows: Why can't I see RootCause Workspaces in the Windows Explorer?

Generally you should be able to -- they should show up with special Icon. If they don't then you're running Windows NT or the Icon wasn't registered right. In any case, workspaces are system folders, so you have to set your Folder Options to uncheck the Hide System Folders option.

11.8 Windows: Something happened during Uninstalling the previous version, now I can't install the new one. What do I do?

Older versions of RC would get "stuck" in a weird state where you can neither Uninstall nor Install. The following is the (ugly) procedure you need to follow to get your current version of RC completely uninstalled, and the newer version installed.

I. Procedure for Manually UnInstalling RootCause

1. Turn Intercept off by entering "rootcause off" in a CMD window.
2. Launch the regedit utility by entering "regedit" in a CMD window.
3. Navigate to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services. Scroll down and you will see a subordinate Key named "rci". Delete this rci" Key.
4. Again in the regedit utility, navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\. Scroll down until you see a subordinate Key named "OC Systems". Delete the OC Systems Key.
5. Reboot your machine.
6. Reboot your machine again. Yes, our device driver guy says we need 2 reboots here.
7. You might want to save a copy of your license key file, it's at %APROBE%\licenses\license.dat.
8. Delete the RootCause installation directory (%APROBE%): e.g. C:\Program Files\OC Systems.
9. Delete the following files: %SYSTEMROOT%\system32\drivers\rci.sys and %SYSTEMROOT%\system32\rcjit.exe.

II. Procedure for Installing the New RootCause

1. Install it. You can skip supplying a license key during installation (just ignore the warning about not supplying a license key).
2. When Installation completes, do _not_ reboot yet.
3. Copy your license key (the one you saved above) into the new installation.
4. Now, re-boot your machine.

11.9 How do I stop tracing something I've got a workspace for?

You need to delete it from the registry. The easiest way to do this is with the GUI:

• Open the workspace in the RootCause Console GUI
• In the RootCause main window, click Unregister Program in the Workspace menu.

To unregister a native program, first do rootcause register -l to see the exact path of the program that is registered, then do rootcause register -d -x exe_path.

11.10 What do I do about the message "(E) ADI checksum (0x84b1c4d) does not match module checksum (0xa1c5e35)." when I register on a .dply file at a remote site?

This message will be followed by specific information about the ADI file and module. The module is the executable or DLL on the remote machine, and the ADI file contains the debug information from the host machine where the workspace was developed. The error messages indicate that the version of the module (application) on the remote machine does not match the version against which you developed your original traces. You must create the workspace and traces against the same version you send tot he remote site because we compare checksums. The only difference is that on Windows, the PDB need not be present on the remote machine because the ADI file contains the information that is needed. Unfortunately on Windows the Visual C++ release build defaults to stripping symbols (on Unix the default is to leave them in). Therefore you need to build to get symbols - you don't need debug, just symbols. If you do want debug in order to support the full range of probes, then you should build add /Zi and /link /debug to the Release build options when you build an application that is to be shipped. This is described here.

11.11 (Windows) Why won't the "RootCause On" button stay checked in the GUI?

If after installing the RootCause Console, you notice that when you click the "rootcause on" button in the main GUI window the checkbox won't stay checked, and when you execute the "rootcause on" command from a CMD window you get the following apparent error message:

(E) The program intercept mechanism is not running.
      This is an installation problem: contact support@ocsystems.com
      RootCause registry found and RootCause is DISABLED.

If a system reboot does not correct the problem, please follow this procedure to obtain debug information to assist OC Systems support in resolving your problem:

1. Open a CMD window and enter "winver". This utility reports what Windows version you are running.
2. Open a CMD window and enter "eventvwr" - this will launch the Event Viewer application (you can also use Control Panel -> Administrator Tools -> Event Viewer).
3. Within Event Viewer, do a "save as" operation on the System Event Log, being sure to save the file as a "Event Log (.evt)" type file.
4. Please forward the output from the "winver" command and the .evt file to support@ocsystems.com

11.12 Why does my Java app fail with "Class Not Found" under RootCause, but work fine without RootCause?

The most likely cause of this is that you're using the "-jar" option on your 'java' command, which is not supported by RootCause prior to version 2.1.2 (October 2003).

So, if your application is run with

java -jar $APROBE/lib/probeit.jar

java -classpath $APROBE/lib/probeit.jar  com.ocsystems.probeit.Main

If you don't know what the main class is, it is defined in the manifest of the .jar file. For instance:

mkdir tmp
cd tmp
jar -xf $APROBE/lib/probeit.jar META-INF/MANIFEST.MF 
grep "Main-Class" META-INF/MANIFEST.MF
     This will give a line "Main-Class: com.ocsystems.probeit.Main".
cd ..
rm -rf tmp

You would do the same thing using your own java command line and jar file in place of the above.

After you have changed the command line, you should then re-run the application and got through the "New Workspace" steps. This time it should work fine.

If this is too much of a hassle, contact support@ocsystems.com about getting a version with -jar support. If you weren't using -jar, or if the problem persists after going through above process, also contact OC Systems support and we can help you debug it.

11.13 How can I probe Java classes loaded with a custom class loader and so not in the CLASSPATH?

You will find that when you use "Open Associated Workspace" it imports only the jars in the class path and and so other classes that might be explicitly loaded do not appear in the Trace Setup. This can be easily remedied.

So long as the class loader follows the standard model for class loader inheritance (e.g. classes loaded by that loader have visibility to classes loaded by the application class loader) this is trivial:

1. From the Main Workspace menu choose the Setup->Class Path menu item o bring up the Class Path dialog.
2. In the Class Path dialog, add the path(s) to the class directories or jar files you will be loading from. Note that this does not have to be where they will be loaded from at runtime. This just gets them into the Trace Setup.

If there is no physical representation of the class available, you can use wildcards:

1. Select the Root Java Module in the Trace Setup;
2. Right click to bring up the context menu;
3. Choose Edit Wildcards to pen the >Edit Wildcards dialog.
4. On the left "Trace" side of the dialog, enter strings like:
   "MyClass::*"
"MyClass::aMethod"
   

11.14 When I have "rootcause on" I sometimes notice that commands piped together (for instance "env|grep MyVariable") can hang for a while before completing. Why is this?

Your home directory (which will be the default disk for the rootcause log) is probably on an NFS disk. When two processes try to lock a file at the same time, one will be halted until the other one is done. However, with NFS it can take a while for the state of the unlock to propagate back, leaving the caller waiting on the lock routine even though the other process has unlocked it. The solution is to set APROBE_HOME to a local disk.

11.15 (Windows) Trying to apply RootCause to a service, I get MessageBox (after a reboot) saying there was a timeout and the Service failed to respond. Why?

The System Control manager (SCM - the process that handles the Services applet) is real picky about the timings for Service start and stop. With Rootcause enabled, we may be delaying the start of the Service just enough to cause SCM to complain.

Can you check to see if the Service aborted? Or better said, is the service Service running after you see the ErrorMessage box. Use Task Manager to determine this - once you get the timeout, the Services applet doesn't report Service status properly. If the Service aborted, please let me know - we may have to excluded it from Intercept.

There's a Registry _VALUE_ that controls the Service timeout:
HKLM\SYSTEM\CurrentControlSet\Control\ServicesPipeTimeout.
This is a REG_DWORD value that probably has value 120000 (which is two minutes).

Try increasing this value (e.g. double it) to see if it address the problem.

11.16 When I add my library to the workspace with Add Dynamic Module and run with RootCause, my application never starts. What's wrong and how can I fix it?

Add Dynamic Module causes a library to be "preloaded" (using the aprobe -dll option) because it's only on program startup that automatic trace configuration can be done. However, some user libraries cannot be preloaded because they rely on some global state being defined which isn't done until the program starts running.

On Unix platforms, this (currently) means you can't trace or do anything else on this module. You're beat unless you can change the library to allow it to be pre-loaded.

However, on Windows there is partial support for probing modules that are loaded after program startup. In particular, you can use custom probes, but you can't use the predefined probes which use the "probe all" feature. We wrote a subset of the trace probe for a customer to use on his dynamically loaded Windows library: dyntrace.apc. Give it a try and/or contact us for help.

11.17 (Windows) The APC compiler fails on the giant APC file generated with apcgen. Now what?

If you just use apcgen module.dll > module.apc you'll get a huge file. This file is translated into ANSI C, then compiled with the native compiler. The capacity of the Visual C++ isn't huge, so that can fail with an error like:

module.apc(102598) : fatal error C1076: compiler limit : internal
 heap limit
 reached; use /Zm to specify a higher limit
 (E) apc could not compile the file module.apc_c.c.

You invoke apc with the "/Zm" option as suggested by adding:
   -compiler /Zm300
on the apc command line, which increases the VC++ compiler heap to 300% of the default maximum.

Alternatively, you can attempt to break up the apc file by-hand, or can generate just a subset of the traces by using the -p and -f options on the apcgen command; Use apcgen -h for brief usage or see Appendix A of the Aprobe user's guide.

11.18 (Windows) Why does our application crash due to a bad return code from CoInitializeSecurity() when running under RootCause?

We don't know, exactly. However, we have identified a simple workaround probe:

#include 
probe thread
{
   probe "CoInitializeSecurity()" in "ole32.dll"
   {
      on_exit
      {
         if ($return == RPC_E_TOO_LATE)
         {
             $return = S_OK;
         }
      }
   }
}

To use this:

1. Put the above in an apc file, say coinit_workaround.apc;
2. compile it with    apc coinit_workaround.apc
3. copy the resulting coinit_workaround.dll file into the workspace.
4. rerun the application under rootcause

11.19 Is there a way to add my own files to a deploy file so they will unpack into the directory created by rootcause register xxx.dply?

A .dply file is just a zip file. You can just use zip (provided with RootCause) to add files to this archive, like:
   zip xxx.dply this.txt, that.class, other.ual

11.20 Why doesn't the pi_demo program doesn't run Linux Fedora Core 3?

Because it was built on an old version of Linux. You can rebuild it from source using the Makefile in that directory, or else load the compatibility package for Fedora: compat-libstdc++-*.i386.rpm.

11.21 Why didn't my trace on Linux didn't log any data?

If your Workspace is being accessed over NFS, this means you're writing the data to APD files over NFS, and Linux has known bugs with this. You really need to have your workspace/APD files on a locally-mounted disk. (Even if it weren't for this bug, logging over NFS is orders of magnitude slower.)

11.22 How can I eliminate "WARNING: Could not create system preferences directory" when I start the RootCause GUI?

If you're seeing something like:

Starting RootCause...
eddea02:/home/essc2/josephw/devenv/cstnd/src
==>Jan 25, 2005 8:26:39 PM java.util.prefs.FileSystemPreferences$2 run
INFO: Created user preferences directory.
Jan 25, 2005 8:26:41 PM java.util.prefs.FileSystemPreferences$3 run
WARNING: Could not create system preferences directory. System
preferences are unusable.

Sun's workaround is to run as 'root' with any Java application once.

(There's also a way to eliminate this entirely, but it requires Java 1.4 and for compatibility reasons, the RootCause GUI is built with Java 1.2.2.)

12.1 What is Aprobe?

Aprobe is a suite of tools and libraries which support dynamic modification and extension of a program by dynamically patching the program executable and/or shared libraries.

A dictionary defines "Probe" as "Device for exploring an otherwise inaccessible place or object." "Aprobe" stands for "Algorithmic Probe". It is hence a tool for exploring your program with the help of user-written algorithmic probes. These probes are installed into your program with the help of OC Systems' patented "dynamic action linking" technology.

A user runs a program with the "aprobe" tool, indicating that certain "probes" are to be patched into the program and executed as the program itself runs.

A "probe" consists of "actions" composed in C, with some special syntax added to indicate where in the program the actions are to be invoked.

There are a number of predefined probes included in Aprobe; there is a tool to generate simple probes directly from a linked or unlinked object file; or the user may easily compose his own probes in a simple extension of the C language.

See also "What is RootCause?"

12.2 What is ProbePak?

The ProbePak was an experiment at introducing users to the power of Aprobe and RootCause by making a subset available for free download. It didn't work out, and ProbePak is no longer supported. See the main page www.ocsystems.com for information on our current products.

12.3 What are some potential uses of Aprobe?

Read more about uses of Aprobe in the Product section of the web site or read the white papers in the Resources section. See also "What are some potential uses of RootCause?"

12.4 How do I get started quickly with Aprobe?

The best way to get started writing probes is to look at examples, and make some small changes.

If you have RootCause and have been using the GUI, you can use the Custom... button in the Trace Setup window to generate a probe, and look at that. If that looks too daunting, or you want a more tutorial approach, try the graduated examples in the examples (or ada_examples ) and demo/Aprobe subdirectories of the Aprobe installation. Check out %APROBE%\Examples\Simple\Readme.txt (Windows) or $APROBE/examples/evaluate/README (Unix).

12.5 Who can use Aprobe?

Technical people who are developing, testing, and maintaining software.

12.6 What different versions of Aprobe are there?

The current version of Aprobe on AIX is 4.4.1; on all other platforms it is 4.3.4b, released in June 2005.

The original version of Aprobe is version 2. for AIX, included as part of OC Systems LegacyAda/OATS product, and in earlier versions of OC Systems "PowerAda" product.. While it shares the "probe" concept with the newer version, the user interface and details of Aprobe Version 2 differ substantially from Versions 3 and 4.

12.7 For which platforms is Aprobe available?

Aprobe is currently available on AIX, Linux (x86), Solaris, and Windows 2000/XP.

The detailed requirements are documented in Chapter 2 of the RootCause User's Guide for Unix or Windows .

12.8 How do I get Aprobe?

E-mail , and we will arrange for you to receive the software.

12.9 What documentation is available for Aprobe?

Aprobe is delivered with a User's Guide in hardcopy, HTML, and PDF formats. The latter two softcopy forms are included in the evaluation version which can be downloaded. The HTML version is available on-line at www.ocsystems.com/sup_ug_index.html .

There are a series of graduated examples that come with their own text documentation in the examples and demo subdirectories of the Aprobe installation. You should read %APROBE%\Examples\Simple\Readme.txt (Windows) or $APROBE/examples/evaluate/README (Unix), and try at least some of the examples under that directory, before trying Aprobe on your own application or looking through this FAQ for answers.

12.10 What tools make up Aprobe?

apcgen - generates APC for some or all functions in the specified object file(s)

apc - compiles and links the specified APC file(s) into a UAL (DLL).

aprobe - runs the specified program after loading and applying patches in the specified UALs.

apformat - formats any data logged in the specified aprobe data (APD) file(s).

These tools are described further in other questions below. A number of additional tools and scripts and for specific situations are also provided. See Appendix A of the Aprobe User's Guide.

12.11 How is Aprobe licensed?

Same as RootCause. See Q1.9 .

12.12 Is there a point-and-click (GUI) interface to Aprobe?

Yes. It's called RootCause. See Q1.1 .

In addition, on Windows, we provide a GUI for Aprobe, which is deprecated but still works. Try the Aprobe menu under the Start menu on your workstation to start it.

Also, Some predefined probes (see Q15. below) include a Java GUI to specify configuration parameters for that probe.

12.13 Can I run Aprobe on any executable program file?

Yes. You can run aprobe (without any probes) on any application at all unless:

• It is a secure application which a debugger doesn't have authority to attach to. In this case you should get a clear explanatory message.
• The application does something very strange like replacing some low-level system routines with its own versions that do something different.
• There's a bug in Aprobe.

If you find that using aprobe causes your application to crash, you should try running aprobe without any probes. If it still crashes, it should be reported as a bug to .

A slightly different question is, "Can I use Aprobe to put probes on any program?" To actually apply probes to a native module, there are three basic requirements:

• Symbols

For Aprobe to do what it does it must be able to figure out where the subroutines you are trying to probe have been linked and loaded. We call this location information "symbols". All symbolic debuggers have the same requirement. See Q12.17 .

The symbols may be as originally added to the application (i.e., not stripped, see Q12.16 ), or they may have been saved separately by Aprobe using apmkadi (see Q13.11 ).

Most programs delivered with the operating system, and off-the-shelf software, are stripped, so you can't use Aprobe directly on the application code, but you can generally probe shared libraries (DLLs) that support them.

If the program uses a mechanism that transfers control other than by the normal call and return mechanism, such as setjmp / longjmp or an unsupported exception mechanism, and there is an active probe at the time of that non-standard transfer of control, the program will likely crash.

Ada and C++ (and Java, but that's a separate issue) support exceptions which are non-standard transfers of control. Each compiler does this in a different way, and must be explicitly supported by the Aprobe runtime. See Q12.15 .

12.14 In what language(s) can my program be written?

Same as for RootCause. See Q1.10

12.15 What compiler(s) must have been used to compile my program?

Same as for RootCause. See Q1.11

12.16 (Unix) How do I tell if a program file is "stripped"?

Use the "file" command, e.g.:

Solaris:

$ file a.out 
a.out: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped
$ file /bin/ls 
/bin/ls: ELF 32-bit MSB executable SPARC Version 1, dynamically linked, stripped 

AIX:

$ file a.out
a.out:      executable (RISC System/6000) or object module not stripped
$ file /bin/ls 
bin/ls:     executable (RISC System/6000) or object module

Linux:

$ file a.out
a.out: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped
$ file /bin/ls
/bin/ls: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped

12.17 How do I tell what symbols a program has available?

apcgen -L will list the Aprobe function symbols in any compiled object module, for example:

apcgen -L C:\WinNT\system32\kernel32.dll
apcgen -L /usr/lib/libc.so
apcgen -L /work/programs/prog.exe

There are other apcgen options such as -m to show "mangled" names and -v to show file names--use apcgen -h for usage.

The RootCause Trace Setup window shows a tree of all the functions organized by module, directory and file, using the same mechanism used by apcgen.

If you want information about data symbols, or want to confirm that a function may actually be probed, you can use the apinfo command, which runs the "info" predefined probe. This only works on executable programs. For example:

apinfo -d /work/programs/prog.exe

will show all the global and file-static data symbols found when prog.exe is loaded by aprobe. There are lots of other options: use apinfo -h to see them. See Q13.7 if you're on Windows and apinfo prints nothing at all.

12.18 What do I do to get symbols in my program?

On Unix, every program has its symbols unless they're explicitly stripped (see Q12.16 ). So, to get symbols in the program, don't run the "strip" command or link with an option that causes the resulting program to be stripped. Shared libraries always have at least global (external) symbols.

On Windows, if you have control of how a program is built you can greatly expand the routines available for probing by adding the "/debug" switch and the "/pdb:none" switch when you link. These switches cause symbol information to be generated and the information to be put in the executable.

On Windows, the Microsoft Visual C++ compiler can produce symbols in two major forms or it can produce no symbol information at all. It can also place the various types of symbols either in the executable itself or in two kinds of separate files. Complicating this issue further is that shared libraries (aka DLLs) can be produced with the entire range of debug information forms available. One good item is that DLLs almost always expose their public interface by name.

Generally software you get from a vendor (COTS) will have no symbol information included with it. However Windows programs are generally broken into a number of DLLs and the interface between DLLs is visible to Aprobe. Thus while you can't probe routines local to Notepad.exe you can probe how Notepad.exe makes calls to CreateFileW in KERNEL32.DLL.

12.19 What do I do to get "debug information" in my program?

This is documented in Chapter 10 of the RootCause User's Guide, "Building a Traceable Application", and in Chapter 3 of the Aprobe User's Guide, but it's summarized here:

• For Unix compilers (except PowerAda) compile with -g.
• With PowerAda you get debug information by default, but you need the PowerAda program library available just as you would for adbg.
• For Windows, compile with /Zi and link with /debug.

In addition to compiling with the right option to generate the debug information, you also must retain that information and have it available where it's supposed to be:

• On Windows, this information is put in a separate PDB file, which should be in the same directory as the module itself.
• For gcc-based compilers, including GNAT, and IBM's C and C++ compilers, debug information is collected at link-time into the executable, and is retained unless you explicitly use the "strip" command.
• The Sun WorkShop (Forte) compilers do not copy the debug to the executable, but leave it in the object files and simply build an index in the executable which gives the path to all the object files. You can force the debug information into the executable by compiling (not just linking) with the -xs flag.
• PowerAda line information is recorded in the executable and can't be stripped, so you don't need any debug information at run-time. However, for `apc' and the RootCause GUI, you need the Ada program library, which must be consistent with the executable and available at the same location recorded in the executable. If the library is moved, you can specify its location with the environment variable APROBE_POWERADA_LIBRARY, for example

  export APROBE_POWERADA_LIBRARY=/builds/old/prog1/adalib

12.20 How do I tell if a program file has "debug information"?

The apcgen command will list those functions that have debug information associated with them:

apcgen -Ld a.out

This should be all you need, but on Unix there are some system utilities that look in the object files themselves that may also be used:

• On Solaris, we provide the readelf utility delivered with Aprobe to determine if there is debug information for a given file or function. The command

  readelf -d a.out | grep "N_FUN.*:F" | awk '{ print $NF }'

• will list all functions for which there is debug information. If you don't see what you're looking for in an executable, and you are using the C or C++ compiler, the debug information may be in the individual object file. Use the above command on the appropriate object file, or, if you're not sure where it is, do:

  readelf -d a.out | grep N_OBJ | awk '{ print $NF }' 

• to see all the object files referenced by the executable.

• Linux uses ELF format also, but the similar utility is objdump :

  objdump -G a.out | grep "N_FUN.*:F" | awk '{ print $NF }'

• (Note that while readelf is available on Linux also, it only shows DWARF debug information which isn't yet supported by Aprobe).

• On AIX, you can use the dump utility with the -t option to dump symbol information, including debug "stab" strings. For example:

  dump -t a.out | grep ":F"

• will show the functions that have debug information.

12.21 What is a "probe"?

A "probe" is a "user action" associated with a specific location in a program. The user action is executed whenever control passes through the location with which it is associated. A "probe" is described in an extension of C called "APC", for example:

probe thread 
{    
  probe "foo"     
  {
    on_entry
    {    
      printf("Entering foo.\n");
    }     
  } 
}

The block following the "on_entry" is the "user action". The syntax surrounding it describes exactly where and when the action should be executed: immediately upon entering function "foo()" in each thread.

12.22 What is a "UAL" (.ual file)?

A UAL is a "User Action Library". It is the output of the "apc" command, and is a shared library consisting of the object code generated from your apc files. Not just any shared library (DLL) may be used as a UAL, and it a UAL may not be renamed after creation, because it has specially-named entry points based on its filename which are called by the Aprobe runtime to perform initialization.

12.23 Why does a Windows UAL file have a file extension of ".dll"?

It has an extension of DLL because it is a regular Windows Dynamic Link Library (DLL). In most cases it could also be named .ual , but there are some cases where the .dll is required by Windows.

12.24 What is "logging"?

With respect to Aprobe, "logging" means "writing data to a file for later analysis" Aprobe provides a built-in logging facility that allows saving raw data in a time and space-efficient way, and using "apformat" to display the logged data later. See "Logging Data" for related questions.

12.25 What is an ".apd" file?

An ".apd" file is one that contains the data generated by a program run under aprobe. These are binary files which are read with the "apformat" tool.

There is always a ".apd" file generated giving aprobe invocation information, even if no "log" statements are executed. If log statements were executed there will be a "-1.apd" file, and maybe "-2.apd" files as well.

12.26 What can't I do if my executable or library doesn't have debug information?

You can't reference source-level information in your probes. It's just like using a source level debugger in this respect, and for the same reason. A good rule is, if the debugger can print the value of a variable x at line 15, then you can do "on_line(15) log($x)" in your probe.

More specifically, you need to specify "-x exe_or_library " on the apc command, and the exe_or_library must contain debugging information, if you use a construct in your probe that cannot be resolved without specific debug information from the program. Such constructs are:

(a) target expressions: names from the probed program preceded by $, or $* ($1, $2 are ok, as are hardware-register references starting with '$$').

and

(b) references to specific source lines;

Note that there are lots of probes you can write; for example, all but one of the predefined probes provided with Aprobe will work fine in the absence of debug information, and the one that does require it (coverage) does so in order to get source line number information.

12.27 Does use of on_line() requires application to be have debug information?

Yes, but things aren't that simple. To build a probe that requires debug information (including line information) the debug must be available when the probe is compiled. However, the debug information can then be stripped and the probe ran against the stripped executable.

For the symbol table, the necessary symbols must be present at runtime, either in the application (or application libraries) or in a .adi file which is generated with the Aprobe tool apmkadi . That tool allows you to capture the symbol table in an internal form and then strip the executable.

Also, PowerAda programs always contain source line information -- this is not considered debug information.

Finally for low-level hacking, you can instrument specific offsets using on_offset.

12.28 What is the maximum number of probes allowed?

For probes you are just limited by paging space. For UALs there is a more practical limit - we limit the total number of modules to 255 and that includes UALs.

12.29 Is there access to C++ private/protected variables?

Yes, if it's in the debug we can see it. We don't look at whether the debug says it's private, protected or public - we just use it.

12.30 Is there any way to attach with Aprobe to a running application?

No. This question is very frequently asked. It sounds great in theory but in practice Aprobe is a tool for tracking problems that have yet to happen, not those that have just happened. There is also quite a bit of work done by Aprobe when an application starts up; often doing this to a running application is as big an issue as re-starting the application.Finally for Java you wouldn't be able to change the classpath to see our classes or intercept classes that have already loaded.

12.31 Is there a way to probe a function for which no symbol is available?

Yes, if you know its address and size, you can define a symbol for it using ap_RecordDynamicFuntionSymbol() in the Aprobe Runtime Library and and then apply probes using the define symbol.

defsym.c

#include 
#include 

static char *image(char *s) 
{
   char *s1 = strdup(s);
   return s1;
}

int main (void)
{
  printf (image("Hello\n"));
  return 0;
}

defsym.apc

//---------------------------------------------------------------------------
// Define Dynamic Function Symbol Example
//
// This is an example of using ap_RecordDynamicFunctionSymbol()
// to define symbols when no debug information is available.
//
// NOTE:  If the offset for symbols is wrong the program will
// likely crash because you will have directed Aprobe to instrument
// the wrong piece of code.
//---------------------------------------------------------------------------

#include "aprobe.h"

// To define your symbols early enough to be instrumented and
// probed, you have to define them from a UAL initalize function.
// The initial part of the name must be InitializeUal_, and the first
// character following that must be lower in the ASCII collating order
// than the first character of the UAL name. '0' is the lowest legal 
// character.

void InitializeUal_0_defsym_apc()
{
   // In this example I just define an alias for the symbol "main"
   // and probe that instead.  You have to know the correct offset
   // and size of the function (though size is not so critical).
   // The offset is the offset in the moudle, not just the text
   // section.
   ap_SymbolIdT NewSym = 
      ap_RecordDynamicFunctionSymbol (
         ap_ApplicationModuleId(),
         "MyAliasForMain",
         ap_ExternSymbol,
         ap_IntegerToOffset(0x10),
         0x1d,
         0);
   if (ap_IsNoSymbolId(NewSym))
   {
      printf("Couldn't define symbol...\n");
   }
}

probe thread
{
   // You'll get a warning about the symbol not being defined
   // when you compile this with apc, but it's OK.
   probe "MyAliasForMain"
   {
      on_entry  printf("Hello again...\n");
   }
}

13.1 What does "aprobe" do?

Aprobe locates the specified UALs (if any), loads them as well as the Aprobe runtime, patches the executable to invoke the probes described in the UAL files, and starts execution of the specified program.

13.2 How do I specify options to my program when using aprobe?

The executable program name is the last argument on the aprobe command line. All options after that are passed as arguments to the executable. For example, if your regular command-line would be:

    mygrep "a_string" *.txt

Then with aprobe it would be, on Unix:

  aprobe -u mygrep.ual mygrep "a_string" *.txt

and on Windows:

  aprobe -u mygrep.dll mygrep.exe \"a_string\" *.txt

Note the backslashes needed to preserve quotes passed to the program.

The most reliable way to do this, used by RootCause, is with the aprobe "-execvp" option. In this case you specify a filename in place of the parameters, and the filename includes all arguments, including "argv[0]" that is to be passed as the executablename. For example, in the above case:

aprobe -execvp -u mygrep.ual mygrep mygrep.args

where mygrep.args might contain the lines:

mygrep.exe
"a_string"
file1.txt
file2.txt

13.3 How do I specify options to my probes?

Options and parameters can be passed to each UAL as well. This is done by following the UAL name with the -p option followed by the options in quotes. This is most commonly seen when invoking a predefined probe that is part of Aprobe, for example:

   aprobe -u info -p "-sa" mygrep.exe

The options to the info probe are "-sa".

13.4 How do I print my output at run time instead of sending to the APD file?

The "-if" ("immediate format") option on the aprobe command does this, e.g.,

   aprobe -if -u fooTest foo

13.5 Can I suppress generating an ".apd" file?

Not at this time. Even if you do "aprobe -if -n 0 ... " you get the basic .apd file.

13.6 How can I run my probes without invoking aprobe?

Use RootCause. That's one of it's key features If for some reason you can't do that, you can

• Substitute the Aprobe command line for the command that starts your program
• Rename your application and replace it with a hard-coded script that calls aprobe on the renamed executable. On Unix there is a script delivered that facilitates this, called "run_with_aprobe_edit", which documents its use.
• On Unix, use the "run_with_aprobe_apo" script. The script text includes documentation on its use.
• On Solaris there's an additional option of linking aprobe into your application. This mechanism is coming soon for AIX and Linux, but will never be available for Windows.

See Chapter 4 of the Aprobe User's Guide, "Loading Probes without aprobe".

13.7 Windows: When I run my program with aprobe I don't get any output even though I know I'm executing routines with probes on them and those probes use printf. What's going on?

First make sure that Aprobe was correctly installed. You can do this by running one of the examples in the Aprobe\examples\... directories.

The most common reason you don't see any output is that standard output from your program is going to the null device.

A windows program can be linked for one of several Windows subsystems. If the subsystem is the "Windows GUI subsystem" standard output seems to go elsewhere. You can determine what subsystem your program has been linked for by using QuickView (look for the "subsystem" entry under the "Image Optional Header"). You can still get your output by redirecting it to a file using the -o switch on the Aprobe command. For example:

   aprobe -u myual.dll -o!stdout.txt!stderr.txt myexe.exe

In this case you are indicating that:

standard input: <none> standard output: stdout.txt standard error: stderr.txt

13.8 Windows: When I use the -o switch to redirect the output of my program to a file(s), the output seems to be out of order?

This is a result of the buffering that Windows does for each executable/DLL.

For each module (the executable and all the DLLs) NT sets up an individual output buffer for standard output and standard error. If the output is going to a device like a command window no buffering is done and the output from different modules is interleaved on the output device. However if the output is going to a file then the buffer is dumped only when it fills (or the program ends). You can control this buffering by using the C runtime call setvbuf() .

Aprobe turns off buffering in each UAL you produce. Thus if you are using multiple UALs and the Aprobe runtime all output will be interleaved correctly. Unfortunately output from your application may still be buffered. Currently your options are to direct stdout and stderr to a terminal device like a command window or you can recompile your application and make the following call in it:

  setvbuf(stdout, NULL, _IONBF, 0);

In the near future we will provide a way in a probe to turn off buffering in your target program.

13.9 Unix: How do I probe a function in a dynamically-loaded shared library?

If your program explicitly loads a file by calling dlopen("dynamic.so"), Aprobe does not support this directly since it does all its patching when the executable and any shared libraries linked in are first loaded into memory. So the only shared libraries you can probe are those listed by the command

ldd exe_name

However, the LD_PRELOAD environment variable can sometimes be used to achieve the same goal. Suppose that we have executable a.out which will load at some point libmyfuncs.so using dlopen. The following would cause the shared library to be loaded with a.out and thus accessible to Aprobe:

LD_PRELOAD="/full/path/libmyfuncs.so" aprobe -u myprobes.ual a.out

This assumes that libmyfuncs.so is not dependent on any other shared libraries and that it doesn't hurt to be initialized earlier than would have been the case with dlopen.

13.10 Can I probe a function in native C or C++ code loaded by a Java application?

In general, if you're using Java, you should be using RootCause and not Aprobe directly. However, you can do this using apjava -dll option. If your Java class named "JTest" contains "LoadLibrary("native") then this should work:

apjava -dll /full/path/libnative.so -u native_probes.ual -java JTest

or similarly, for Windows

apjava -dll \myprograms\native.dll -u native_probes.dll -java JTest

13.11 Is there a way I can use Aprobe in a target environment where my application has no symbol or debug information with it (is stripped)?

If you have a program that can be probed, you can run the tool apmkadi on it to create an Aprobe Debug Information (ADI) file. You can then remove the symbols from the executable (using the strip command on Unix, or removing the PDB file on Windows) and ship it to the target site. When you want to run Aprobe on that, you would then specify not only the UAL file(s) containing the probes, but the ADI file(s) as well, which contain only the symbolic information needed by Aprobe. See Appendix A, "apmkadi" for more information.

13.12 Can I run aprobe but produce no APD files?

Yes. The "-p" flag, which prevents generation of any APD files, was introduced in Aprobe version 4.2.5. This is useful if your probes don't log any data using the default log method.

13.13 Why does my program crash when using aprobe, and not without?

The possibilities are:

1. There's a bug in your probe, for example, one of your action routines is dereferencing a null pointer. See "Debugging Your Probes" near the end of Chapter 3 of the Aprobe User's Guide.
2. (Unix only) Your application provides its own "malloc()" function which requires initialization before its first use. Since aprobe gets control before your application does, and uses the application's malloc(), this could cause a crash on startup. See also Q15.14.
3. (Windows only) The probe is opening a file to access symbol information when the application is in a state where this is not allowed. Since such states are not documented, aprobe has a "-R" flag that forces all symbol information to be loaded before the program starts. Using "aprobe -R" makes for slower startup on big applications but avoids this apparent Windows bug.
4. Your probe is accessing or logging data on_exit to a function or thread, but the on_exit action is being called in an exception or thread exit condition and so may not have valid data available. In order to check for this, put on_exit code within a block that checks the ap_ProbeActionReason implicit parameter, e.g.:

     on_exit {
       if (ap_ProbeActionReason == ap_ExitAction)
       {
         log("foo returns ", $return);
       }
       else
       {
         log("foo exits abnormally for: ", ap_ProbeActionReason);
       }
     }

5. Your program is very time-critical, or is such that timing may change the order in which order-dependent operations are executed. Aprobe introduces some overhead, and your probes likely introduce a lot more overhead, which can change your program's behavior. You can use aprobe itself to find out what's happening, and to force synchronization between threads -- contact .
6. There's an Aprobe capacity problem. This may happen with the predefined probes if you select all functions, or (equivalently) specify "*" IN "*" in the configuration file. (See Q. 4.9). You can either reduce the number of functions you're probing, or increase the default probe stack size with "aprobe -q stacksize=20000000" (or some other big number).
7. You're probing a function that doesn't follow standard generated code conventions. This can happen when you try to probe everything in a shared system library such as ntdll.dll on Windows, libc.so on Solaris or Linux, or libc.a(shr.o) on AIX.. See if it reproduces when you only probe known entry points in the system library, or limit your probes to your application module only.
8. There's a bug in Aprobe. Contact Contact our sales department for more information ( ).

13.14 AIX: Aprobe version 3.2 had the -s1 option to prevent conflicts with my application's shared memory. Is there a similar feature in version 4.2?

We hoped that by getting rid of shmat() from our code that we would no longer cause conflicts. Unfortunately we didn't realize that the OS would choose memory map addresses that would conflict, so the problem immediately reappeared. We added a different flag to allow you to specify the memory area that should be used: -q mmap=address where address is the address that should be passed to mmap() when Aprobe requests its shared memory. For example:
aprobe -q mmap=0xd0000000 -u myprobes myapp.exe

If you don't have this flag, you'll need an updated version of Aprobe but you might be able to get around it:

Many users find that they can avoid shared memory conflicts simply reducing the size of the APD files. The default maximum size is 256M persistent and 256M user APD file. By using a ring (aprobe -n flag) you can vastly reduce the user apd size and you can use the -sp flag to specify a reduced persistent file. For instance, the following:
aprobe -sp 16000000 -n 5

will create a persistent file of approx 16M and up to 5 APD files of 2M each.

13.15 Why does Aprobe ask for such a large memory-mapped file on startup, when I've specified only a 4M APD file with "-s"?

The size of the persistent APD file is controlled independently of the size of the APD ring files. You can use the -sp option to lower this significantly. The default is 256Mbytes because we need to set it to the maximum at the beginning. However we've found that 16M is generally sufficient in practice.

If you look and see how big your persistent files grow you can use that at a baseline. The main things that get logged to the persistent file after program start are:

• New java classes / methods
• b) New threads
• Tracebacks recorded with the traceback to ID mechanism
• LOAD_SHED functions

13.16 On Solaris and/or Linux, when I run my application under Aprobe it crashes during initialization with a problem in malloc. This doesn't happen without Aprobe. Why?

The application might have a poor implementation of malloc built-in. On Solaris and Linux an application can provide it's own implementation of malloc, free, etc. and this will be used. Most local versions of malloc are well behaved. Some, however, require initializing by the application before first use. Since Aprobe gets in earlier than the main() this can cause a malloc request to be made ahead of it being initialized.

If you have control over the code you should fix this by making the malloc self-initializing. If you don't then, unfortunately, you will not be able to run the application under Aprobe.

13.17 Why can't RootCause see program symbols on a system using Visual Studio .NET 2003?

If (for example) in the Trace Setup GUI window for the pi_demo example, you don't see anything under the pi_demo module, or you don't see all the information described in the demo, then it's likely that RootCause is not finding the DLL it needs to read the debug info.

RootCause uses a DLL that is distributed with Microsoft Visual Studio to read PDB file information. The name of this DLL is slightly different for different versions of Visual Studio. e.g. For VC++ 6.0 it's mspdb60.dll, for .NET (VC++ 7.0)=20 it's mspdb71.dll.

The pi_demo example was built on VC++ 6.0 and by default RootCause will not use the VC++ 7.0 DLL to read VC++ 6.0 PDB files. There's an Environment Variable that can be set to alter this behavior.

Here's what you need to do:

1. Set Environment Variable "APROBE_USE_DIA" to the value "1" either user/system-wide using the System Properties -> Environment Variables applet, or in the CMD window where you launch the RootCause GUI (via the "rootcause open" comand) and the pi_demo application (e.g. enter "set APROBE_USE_DIA=1").
2. Make sure that the path to the location of file mspdb71.dll is in your %PATH% Environment Variable. The easiest way to do this is to execute the vcvars32.bat setup file (under the Visual Studio installation directory) in the CMD window where you launch the RootCause GUI (via "rootcause open") and the pi_demo application.

FYI, the following are the locations of the vcvars32.bat and mspdb71.dll
files for a typical .NET Visual Studio installation:

c:\Program Files\microsoft Visual Studio .NET 2003\VC7\bin\vcvars32.bat
c:\Program Files\Microsoft Visual Studio .NET 2003\Common7\IDE\mspdb71.dll

14.1 What does apformat do?

apformat reads one or more related APD (.apd) files and formats the data they contain. For example, if the command

    aprobe -u a.ual a.exe

produced the files

    a.apd a-1.apd

Then the command

    apformat a.apd

1. Reads a.apd to find out the executable (a.out) and UAL(s) (a.ual or a.dll) that were used by aprobe to generate the file, and what other APD files were generated (a-1.apd).
2. Reads the data records contained in a-1.apd, and for each one, invokes the associated format routine contained in the UAL file, passing the data in the record as parameters to the format routine.

14.2 Which of the ".apd" files do I specify on the command-line?

If you specify the "base" one, without any number at the end (e.g., a.apd), all of the files that were written to during the most recent invocation will be formatted. If you specify an individual data file, such as "a-2.apd", only the data in that specific APD file will be formatted.

14.3 Can I restrict the apformat output to just that generated by one of the several UALs provided at aprobe time?

Yes. Use the "-z" option to indicate that no UALs are to be loaded implicitly, then use "-u" to explicitly state which one you want to use:

  apformat -z -u first myprog.apd

14.4 Can I restrict the apformat output to just that generated by one or two of my format routines?

Yes. If you provided your own format routines, you can do it by editing those routines and re-generating the UAL of the same name as the original .

Lets say you have "dumpall.apc", from which you generated "dumpall.ual". Copy "dumpall.apc" to "dumpall.apc.save". Then edit "dumpall.apc" and comment out the bodies of all the format routines except for the one(s) you want to keep. Use `apc' to compile "dumpall.apc" into "dumpall.ual", e.g., apc dumpall.apc -x myprog then do:

  apformat -z -u dumpall myprog.apd

The UAL name must be preserved because the basename of each UAL is part of the "key" used to map formats to data in the APD.

14.5 Can I programmatically filter which formats are used?

Yes, and this is actually preferable:

1. Define global flags corresponding to the different kinds of filtering you want, initializing all to (say) "true".
2. Code your format routines such that each has "if (FormatFlag1 || FormatFlag2) { ... }" guarding the execution of the print actions in the format routine.
3. In the "on_entry" part of a "probe format", read the command-line arguments to the UAL (ap_UalArgc, ap_UalArgv), or an environment variable, or file, or whatever, to determine the desired settings of the flags.

14.6 Can I do the previous 2 if I'm using automatically generated formats?

No. The formats are generated automatically and there's no way to put your own conditions within them. (Of course you can put conditions around the log statement at run time, so that no data is recorded to begin with, but this is a different issue.)

14.7 When do I need to specify the UAL file to apformat?

When you want to use UALs different from, or in addition to, the ones that were specified when you ran aprobe. You might want to do this in order to only process part of the data, or use different format routines. Use apformat -z if you want to use only those UALs explicitly specified on the apformat command line.

14.8 Can I use "apformat" without an APD file?

No. There must be a valid APD file generated by aprobe.

14.9 Aprobe works fine, but I get a crash from apformat; why?

This is almost certainly because there's a bug in one of your format routines. See "Debugging Your Probes" near the end of Chapter 3 of the Aprobe User's Guide for Unix or Windows .

However, if you didn't write any of your own format routines, either because you're using a predefined probe, or because you just used "log(something);", then this is probably OC Systems' fault and you should contact .

14.10 Can can I use ap_UalArgv in "probe format ... on_entry" to get arguments passed at run-time (aprobe time)?

No. ap_UalArgv at apformat time is for reading arguments passed to the UAL on the apformat command line, as in:

   apformat -u my_probe -p "param1 param2" t.apd

You would have to log the data you need from run-time yourself, and format it later. This can be done by including the following APC file into your APC file prior to the "probe format" or other format routine in which you want to use the arguments. You can then use the variables ap_RuntimeUalArgc and ap_RuntimeUalArgv just as you would use ap_UalArgc/v at run time.

/* logualargs.apc
 * Include this once per UAL to record runtime arguments for format time use. 
 */

#ifndef _LOGUALARGS_APC_
#define _LOGUALARGS_APC_

static int ap_RuntimeUalArgc = 0;
static ap_NameT *ap_RuntimeUalArgv = NULL;
static void ap_RuntimeUalArgStart(ap_Uint32 *argc)
{
   ap_SizeT size = ((*argc)+1) * sizeof(ap_NameT);
   ap_RuntimeUalArgc = *argc; 
   ap_RuntimeUalArgv = (ap_NameT*)(ap_Malloc(size));
   memset(ap_RuntimeUalArgv, 0, size);
}

static void ap_RuntimeUalArgAdd(int *pos, ap_NameT Arg)
{
   ap_RuntimeUalArgv[*pos] = ap_StrDup(Arg);
}

probe program
{
   on_entry
   {
       int i;
       log (ap_UalArgc) 
          with ap_RuntimeUalArgStart to ap_PersistentLogMethod;
       for (i = 0; i < ap_UalArgc; i++)
       {
          log(i, ap_StringValue(ap_UalArgv[i])) 
           with ap_RuntimeUalArgAdd to ap_PersistentLogMethod;
       }
   }
}
#endif

For example:

#include "logualargs.apc"

probe thread 
{
}

probe format
{
   on_entry
   {
       int i;
       // Run-time arguments to this UAL
       printf("ap_RuntimeUalArgc = %d\n", ap_RuntimeUalArgc);
       for (i = 0; i < ap_RuntimeUalArgc; i++)
       {
          printf("ap_RuntimeUalArgv[%d] = \"%s\"\n", i, ap_RuntimeUalArgv[i]);
       }
       // Format-time arguments to this UAL 
      for (i = 0; i < ap_UalArgc; i++)
       {
          printf("ap_UalArgv[%d] = \"%s\"\n", i, ap_UalArgv[i]);
       }
   }
}

15.1 What is a predefined probe?

This is just a UAL containing probes written by OC Systems for a specific purpose. They are generally more complex than ones you would write yourself, and are designed to work on any program that can be probed. Most of these probes include a Java GUI to simplify parameterization of the probe for your specific program, such as specifying the functions to be probed.

All predefined probes are in $APROBE/ual_lib/*.ual; the source code is $APROBE/probes/*.apc. The documentation for these probes is in Appendix D of the User's Guide.

15.2 Do I have to use "apc" to build these probes myself?

No! The UALs for all of the predefined probes are already built and located in $APROBE/ual_lib. This is in the UAL search path, so the simple name of the UAL is sufficient. For example:

    aprobe -u info myprog.exe

15.3 The examples show invocation of predefined probes using aprobe -u info myprog.exe. How does aprobe find these UALs when they're not in the current directory?

The directory $APROBE/ual_lib is always searched for UALs after the working directory. The environment variable APROBE_LIBPATH may also be defined to add additional directories.

15.4 Can I use Coverage without using the Java configuration GUI?

Yes. In fact, that's the default. There is no GUI for `info'. The coverage, profile and trace probes provide a GUI to assist in building or modifying configuration file which defines what should be done, but this file is just a text file that can be edited by hand.

The `memwatch' predefined probe provides a "runtime" GUI to monitor memory usage as the program is running, and to take interactive snapshots of the allocation data.

See the documentation for each probe in Appendix D of the Aprobe User's Guide for Unix or Windows .

15.5 The trace probe really slows down the program--how can I speed it up?

You should see the Aprobe User's Guide documentation about this probe. However, you can try these things in this order:

1. Use Load Shedding by specifying "LoadShedThreshold 10" in your configuration file.
2. Don't use wildcards like "Trace *", but rather use apcgen -L to list specific functions you want to trace and just name those.
3. Use the TRIGGER configuration parameter to specify a specific call-tree you want to trace.
4. Use the circular-buffer mechanism, by specifying SaveTraceDataTo CIRCULAR_BUFFER in the configuration file, rather than logging data in real time. Note that your program must complete in a well-behaved way in order to get a snapshot of the data logged to the circular buffer.

15.6 Unix: How can I get a snapshot of my predefined probe data before my program dumps core?

The ability to take a snapshot when an unexpected signal occurs is provided by combining the predefined probe of your choice with the "sigsegv" probe:

   // my_coverage.apc
   #include "sigsegv.h"
   #include "coverage.h"
   static void MyHandler(int sig, void *Data)
   {
      ap_Coverage_DoSnapshot("Snapshot on signal.");
   }
   probe program
   {
      on_entry
      {
         ap_Sigsegv_AddCallback(MyHandler, NULL);
      }
   }   

  $ apc my_coverage.apc coverage.ual sigsegv.ual # creates my_coverage.ual

15.7 Is there a way to invoke predefined probe operations from within my probes?

An API for each predefined probe is defined by the ".h" file corresponding to it in $APROBE/probes. For example, "profile.h" defines "ap_Profile_DoSnapshotForAll()". To call this, you would #include "profile.h" in your APC file (it's in $APROBE/include as well, which is always searched for include files). Then when you compile your apc file, specify the UAL as if it were just another object file to link with:

    apc myprofile.apc profile.ual (Windows
: apc myprofile.apc profile.lib)

This will produce myprofile.ual (or myprofile.dll on Windows).

15.8 How can my probes use the Java GUI facilities that the predefined probes use?

There are two interfaces to the Java GUI objects used by the predefined probes. The one to start with is defined in $APROBE/include/quick_gui.h and implemented in quick_gui.ual( quick_gui.dll on Windows ) . This supports simple graphs, and interactive message, Yes/No, and confirmation dialogs. An example of using this is given in the example $APROBE/examples/learn/visualize_data/ .

The full GUI interface used by the predefined probes like profile.ual is apGUI.h , but this is only for fearless experts.

15.9 I'd like to customize a predefined probe -- how do I rebuild it?

On Windows, simply edit the files in %APROBE%\probes , then

cd %APROBE%\ual_lib
nmake

On Unix you'll probably need to copy them locally, which is a bit ugly:

mkdir my_aprobe ; cd my_aprobe
cd my_aprobe
ln -s $APROBE/include $APROBE/lib $APROBE/bin .
mkdir ual_lib
mkdir probes
cd probes
cp $APROBE/probes/memwatch.apc . # if you wanted to edit memwatch
ln -s $APROBE/probes/* .         # to get everything else
chmod +w memwatch.apc
# edit memwatch.apc (or whatever) as desired
cd ../ual_lib
make -f $APROBE/ual_lib/Makefile memwatch.ual # or whatever

If you have problems or questions, contact .

15.10 How do I use the coverage probe with multiple test cases?

The `atcmerge' tool merges formatted results from different runs on the same or different executables. You can use the aprobe "-d" option to create different APD filesets and corresponding ".tc" files for each run, and use the "atcmerge" tool to merge these. See Aprobe\Examples\Advanced\Test_Coverage for an example.

15.11 Where did the "heap" probe go?

heap.ual (heap.dll on Windows) has been superseded by memwatch.ual. This is a simpler, more robust probe that provides information about allocation patterns, but does not save all the additional data necessary to do error checking. Contact OC Systems if you need a probe with this allocation-checking functionality.

15.12 How do I use this "events" probe everyone's talking about?

With RootCause 2.0.5 (Aprobe 4.2.5) there's an example under examples/predefined_probes/events (Windows: Examples\Predefined\Events), and documentation in Appendix D of the Aprobe User's Guide. Here's a quick summary we sent to a user:

You must have an app_name .events.cfg file, otherwise events does nothing. Let's take a simple case with the routines one() and two() which both call routine three() which, in turn, calls routine four():

   main()
      one()
         three()
            four()
      two()
         three()
            four()

The simplest configuration file is:

EVENT FUNCTION one()
EVENT FUNCTION two()
EVENT FUNCTION three()
EVENT FUNCTION four()

To just look at the calls nested under one() you would add:

FOCUS one()

If you wanted to restrict this at runtime:

FOCUS RUNTIME one()

Let's say that the processing for one() becomes more complex and you want to do end in another routine. This would do the trick:

EVENT START MyEvent one() ON ENTRY
EVENT START MyEvent another() ON_ENTRY

FOCUS MyEvent
FOCUS RUNTIME MyEvent

15.13 In the `profile' probe, what do "Calls to Self/Child" columns mean?

Assume we have a program foo with two functions outer() and inner() . outer loops and calls inner which does some work. We setup the foo.profile.cfg file to profile both of them.

If we look at the output for routine outer we would expect to see Calls to Self being one - it's just called once. Calls to Child should be something like 10 or however many times inner is called.

Similarly the two tables show individual and cumulative time. The individual time for outer would be much lower than the cumulative time since the individual time has all of the recorded times for inner subtracted from it.

Finally, note that this only applies to routines profiled. If outer also calls routine another() which is not profiled, another 's call counts do not show and its time is recorded as part of outer 's individual time.

15.14 Why don't memstat, memwatch, heap probes work on my application?

The most likely reason is that your application doesn't use the default system allocation routines. These might be actual replacements for malloc(), etc. in your own application or in another library such as libsafe or libefence.

Sometimes if you explicitly replace malloc() it can break RootCause/Aprobe completely: see Q13.13.

If Aprobe mostly works except for memory probes, then you can override the default routines used by memwatch by registering for your own allocation routines, or by changing the probe itself. This will require writing or editing some apc code, depending on your exact situation. for further assistance.

15.15 Can you please explain the fields "Alloc Count" and "Free Count" in the memstat "Outstanding Allocation" report?

A specific allocation point (see below) might be reached just once (usually at initialization) and will have an Alloc Count of 1. It may or may not ever free that so the Free Count will be 0 or 1. But many (most!) applications have allocation points that give rise to more than one allocation. For instance:

for (i = 0; i < 10; i++)
{
   linkedList.add (new MyObject (i));
}

Obviously each instance of MyObject was created from the same allocation point. Most growth happens this way - in fact we don't count any allocations we only see once as growth.

What is an allocation point? For native code it's the unique traceback up to the current maximum depth, something like:

  Line 10 of a()
  called from Line 15 of b()
  called from Line 32 of c()

For Java each allocation point is a combination of a traceback and the object type allocated there.

15.16 Can I use memstat to track all allocations and frees?

The default setting of the memstat probes is to pinpoint leaks in a longer-running program. However, you can change the options. From the main RC window select the memstat probe in the UAL list, right click and choose Edit UAL. From the Runtime tab change the Sampling Ratio to 1 so you see every allocation.

From the Format tab check the Display Freed Allocations box. You might also find the Display Zero Growth Allocations useful. Next run, you'll start seeing those freed allocations.

Click the OK button and then the Build button. Re-format (either through the Index or Examine button) and the reports should have the information you need.

15.17 Is there a way to only report allocations in a certain module based on the stack traceback entries?

This mechanism wasn't available in memstat until version 2.1.4b (June 2005), (only in memwatch, which is more focused on individual allocations). For earlier versions, you could edit and build your own custom version of "combined_memstat.apc" that has filtering: see filtermemory.apc.

Version 2.1.4b also introduced EXCLUDE filters in memstat and memwatch, which eliminate the named stack traces and show all others. See $APROBE/probes/[java]_memstat.cfg or $APROBE/probes/memwatch.cfg for usage information.

15.18 Is there a predefined probe for detecting memory corruption?

Yes. The "memcheck" probe, introduced in version 2.1.3 (February 2004) uses a "fence" mechanism to detect corruption of allocated (but not stack/local) memory. It also reports double deallocations.

15.19 Is there a predefined probe for tracking down lock contentions?

We have done some work in this area for customers, but we have not productized it, because the platform- and problem-specifics are not easily generalized. If you want some unsupported probes to start from please contact us.

15.20 What options in the trace.cfg file are obsolete, and why?

Many changes have occurred as the Trace predefined probe has been adapted to support RootCause users. A number of the options have been deprecated, and others apply only when used directly outside of RootCause.

The following options have been deprecated.

MaxDepthOfTracedCalls, DefaultLevels

These were synonyms. It is no longer possible to specify a maximum depth at which tracing is disabled.

LogTimes

times are always logged.

LogLines

lines are logged if and only if specified on each TRACE line with LINES.

TracingEnabledInitially

Tracing is enabled initially if and only if no TRIGGER lines appear. if there are one or more TRIGGER lines, then tracing is only enabled when executing the functions specified by the TRIGGER(s).

CallCountOptions. ExactCallCounts

call counts are now done at format-time by the RC trace display rather than by the trace probe, so these options don't apply.

IndexSymbols

Symbols cannot be indexed.

MaxIndentLevelsBeforeWrap, IndentColumns, AlwaysShowNumericNestingLevel

these used to control formatting but now custom formatting is done by providing alternative formatting routines. The ones provided for the RootCause Trace Display are in $APROBE/probes/rc_formats.ual.

15.21 How do I force a snapshot from a predefined probe?

   $  dbx -a 12345
   (dbx) call ap_Statprof_Snapshot( "dbx" );
   (dbx) detach
   $  apformat appdriver.apd

  // my_profile.apc
  #include "profile.h"
  probe thread {
    probe "abnormal_end_signal_was_handled" {
      on_entry ap_Profile_DoSnapshotForAll( "probe snap",  FALSE );
    }
  }

  $ apc my_profile.apc profile.ual # creates my_profile.ual

A special case of this is to take a snapshot when an unexpected signal occurs: see Q15.6.

15.21 Why does the memstat summary file say it can't do the analysis because I only have one sample?

Some possibilities are:

• You didn't run for long enough. A couple of minutes isn't long enough if you want to run the statistical sampling.
• You didn't format all of the available data. You can do this by selecting all of the apd files for the ring instead of the default (which is just the last one).
• You don't have a real problem. This is more common than you think: People often see instability that they think are memory leakage issues that aren't.

15.22 Could you explain the memstat summary's "Leaked Memory" and "Total Leakage" values?

The statistical part works like this. Say you have a setting (Sampling rate) of one in thirty. Every 30th allocation we record it in a table. Every free gets looked up in that table. If it is in there it is recorded, if it isn't it is ignored. So the sampling is only on the allocations, not the frees.

In the table, the totals (including leaked memory) and counts are multiplied by the sampling rate. If you have enough samples, this will be entirely valid.

We record what you pass to the O/S, not necessarily what the O/S actually allocates. This could under-estimate the amount of memory in certain cases. (e.g. if the memory manager always allocates in quad-word steps it would allocate 16 bytes when you requested 4).

The statistics that identify certain allocation points as "Growth" are based on least squares linear regression analysis.

15.23 How can I define a memstat (or memwatch) filter matching any number of call levels?

That is, is there a way to do something like the following?

   FILTER      extern:"malloc_y_heap()" in "libc.a(shr.o)"
        ==> **** any number of levels matching anything **** 
        ==> "ap_demangle.c":"Demangle_Xlc_Symbol_Name()" at line 2103 (ap_demangle.c)

No, the best you can do is enumerate all the possible matches from your test cases. Wildcards of one or more levels may be implemented in the future.

15.24 Is there a predefined probe to check for stack corruption?

Not officially, but we have written stackcheck.apc for a customer. This version is just for Windows, and checks that the return address is not corrupted on_entry and on_exit to all instrumented functions. instrumentation is hard-coded in the probe for now. A configuration file or separate cconfiguration probe could be added to handle specifying the instrumentation points.

16.1 What does apc do?

The apc command translates one or more APC files into C, and then uses a native C compiler to compile these into object code, and link them with other files specified on the command-line to form a shared library called a UAL. A UAL has a suffix of .ual on Unix, but .dll on Windows due to limitations in how dynamically-loaded libraries are selected on Windows.

16.2 How do I indicate what C compiler and options apc should use?

On Windows, only one compiler is supported, so the C compiler is simply your installed version of Microsoft Visual C++. On Unix, the compiler is defined in the file $APROBE/lib/compiler_profiles and by the APROBE_CC_COMMAND environment variable. This is described in the Files Reference (Appendix B) of the Aprobe User's Guide.

Options to the compiler can also be specified on the aprobe command line by including them in quotes after the "-compiler" option, for example,

    apc foo.apc -compiler "-v"

16.3 Do I need to specify an object file or executable to apc?

You need to specify "-x object module" if you use a construct in your APC that cannot be resolved without specific symbol table or debug information from the program. Such constructs are:

1. target expressions: names from the probed program preceded by `$', or "$*" ($1, $2 are ok, as are hardware-register references starting with '$$'), and
2. references to specific source lines.

In general, probes that you compose to gather information about specific parts of your program will contain one of the above, and you'll want to include the executable or an object file.

For probes on shared libraries which don't contain any debug information, or for probes that should apply to any program (like the predefine probes included with Aprobe), you generally will not provide an object module.

16.4 How do I specify other object files to link into my UAL?

Just include them on the apc command-line. Linker options are specified in quotes after the "-linker" flag, for example,

    apc foo.apc -linker "-lX11"

or, on Windows:

    apc foo.apc -linker "/WARN"

16.5 apc says my function name's not known--why not?

There are a number of possibilities. If you specified "-x ... " on the apc command line, then it means it couldn't find the named function in that file's symbol table. Since apc works pretty hard to match incomplete function names, the name is probably wrong in case or spelling, or, if you provided a parameter profile, it's probably not exactly what the C++ compiler encoded as the name for the function.

You could try using apcgen to generate a probe template for all the functions in the source file (or object file, if it's a template instance) containing the function you want, or the tool apinfo or apsymbols to dump out all the function names in the whole program.

16.6 Solaris: Where can I download a good gcc installation to use with RootCause?

For Solaris platforms, we recommend downloading from http://www.sunfreeware.com.

At this site's home page you will see on the right a list of processor/Operating systems combinations. Click the one which is appropriate for your system. (Note that this list includes both SPARC and Intel--be sure to select a SPARC download.)

Below the processor/OS list will be a list of software packages. Select gcc-2.95.3 (RootCause does not yet fully support gcc version 3).

The link to the binary gcc installation will appear in the center pane of your browser. Download the gcc 2.95.3 image from here. Use gunzip to uncompress the file, then use pkgadd to install the package (it will go under /usr/local). You will need root authority to do this:

pkgadd -d gcc-2.95.3-sol7-sparc-local

16.7 How do I generate debug information for my APC files so line and function information show up in tracebacks?

As with C, use the -g flag; this passes the appropriate debug options to the C compiler (even on Windows: /Zi /Yd /FAcs) and saves the generated C source file.

16.8 (Unix)Can I specify an environment variable for the compiler path in the compiler_profiles file?

Yes! If "ls -l ${CC_PATH}/bin/gcc" on the command-line shows that the compiler exists, then a stanza like:
CC_COMMAND ${CC_PATH}/bin/gcc
will work.

Also note that the environment variable APROBE_COMPILER_PROFILES can be used to override the default of $APROBE/lib/compiler_profiles and point to your own variant of this file. See compiler_profiles file in the user's guide.

16.9 How do I compile a probe for a 32-bit app when running 64-bit Linux?

If you build your application with the compilation option -m32, then to build your probe you'll need to pass -m32 to apc's backend compiler, plus define the i386 macro to the preprocessor. For example:
   apc -Di396 -compiler -m32 -linker -melf_i386 foo.apc
The link stage just invokes ld directly which should automatically build a 32-bit shared library from a 32-bit object file.

If you're going to be doing this regularly you should edit $APROBE/lib/compiler_profiles to update the CFLAGS and PREPROCESS lines so these options are applied automatically.

17.1 How do I use "apcgen" to generate a probe automatically?

You need an object file or executable that contains debug information, i.e., was compiled with debug (see Q12.19 ) or a C header file. For example:

   apcgen foo.exe > foo.apc
  apc foo.apc -x foo.exe

generates foo.apc, an APC file probing all the user-defined functions in foo.exe that have debug information, then compiles that into a UAL.

  apcgen -qparams -p sin -o math_sin.apc /usr/include/math.h
  apc math_sin.apc -x /usr/include/math.h

generates and compiles math_sin.apc containing a probe on the sin() function which logs the parameter and return value. Use apcgen -h to see what options are available to control the output.

Note that RootCause provides this functionality in a point-and-click GUI.

17.2 How do I write a "probe"?

One way is to start with a file generated by "apcgen" (see previous Q.). Or you compose one in your favorite text editor. It's pretty much like writing C, but there's some syntax needed to indicate where and when your probe should be executed. Here's a very simple one:

  probe thread     
  {
    probe "main"
    {           
      on_entry            
      {               
        printf("Entering main.\n");
      }
    }     
  }

If you put this in the file "foo.apc", then you would compile it:

   apc foo.apc 

which produces "foo.ual" (foo.dll on Windows), which you can then probe your program with:

   aprobe -u foo foo.exe

17.3 What is the difference between APC and straight C?

There are several differences:

1) There is special syntax to indicate where and when the probe should be executed, such as "probe", "on_entry", "on_exit", "on_line", etc.

2) There is a special keyword called "log" for recording data at run time and defining the format with which it should be displayed afterward.

3) There are special data references, called "target expressions" which start with `$' and refer to values in the probed program.

All of these are expanded or converted to ANSI C by the apc compiler.

In addition, there is an implicit " #include "aprobe.h ", which makes available the extensive Aprobe API defined in APROBE/include/aprobe.h and documented in Appendix C of the Aprobe User's Guide.

17.4 Why do I need a "probe thread"?

This is an artifact of the clever Aprobe scoping rules. When one probe is nested within another (that is, defined in the declarative part of an enclosing probe), it not only gives visibility to the enclosing probe's data as you would expect, it also means that the inner probe is "active" (its actions may be executed) only if the outer probe is active.

Since every function is executed within some thread of execution, if a function probe weren't inside a thread probe it would never be active.

Anyway, just put in the probe thread{ .. }. It's what works.

17.5 What's the difference between "probe thread" and "probe program"?

The on_entry actions of a "probe program" occur once each, before calling main() (or WinMain() , etc.) and after returning from main() , respectively. The corresponding actions of a "probe thread" occur at the creation and destruction of each separate thread.

Data defined in the declarative part of a "probe thread" is global to all probes, but is unique for each thread. There is always at least one, the "main" thread, which is conceptually nested immediately within the probe program.

17.6 When exactly are the "on_entry" and "on_exit" parts of a function probe executed?

On AIX, Linux and Windows, the on_entry actions are executed before the first instruction of the function itself. In particular, the function's local stack frame hasn't been created yet.

The on_entry actions are executed immediately after the SPARC save instruction has shifted the register window yet, but before any compiler-generated saves of parameters or other values.

The on_exit actions are executed after the stack frame has been discarded, so local data is not available. The next (target program) instruction executed will be the one following the call to the probed function.

17.7 Why can't I dump some parameters in the on_exit part?

Parameters passed by value are essentially local data. They are stored on the stack and the stack frame has been discarded by the time the on_exit part is executed.

If you want to be able to access the input parameters you can save them in the on_entry part, for example:

probe thread 
{   
  probe "foo"   
  {      
    int parm1;

     on_entry      
    {
      parm1 = $1;      
    }      

    on_exit      
    {
      if (parm1 == 1)
      {
         ...
      }      
    }   
  } 
}

C++ reference parameters, and composite parameters passed by reference to Ada, are available by-name on_exit because `apc' implicitly generates code in an on_entry section to save the address passed in. GNAT Ada OUT and IN OUT parameters can be displayed because these are implemented as fields of a 'struct' returned by the function.

17.8 Why is my local variable "unknown" in on_entry and on_exit parts?

The on_entry and on_exit parts are conceptually outside the scope of the function, so the local data is not visible. Local data is visible only within an "on_line" action.

17.9 Is there a way to probe "the first line" or "the last line" in my function?

Yes. Simply write on_line(first) or on_line(last) . You can use this to do function-relative line numbers as well, such as on_line(first+5) .

17.10 How do I specify which of several overloaded functions I want to probe?

In C++, you must specify the exact parameter profile encoded in symbol table by the C++ compiler. The best way to get this is either to look at the output of "apcgen -vL" applied to the object file generated by the compiler, or use ` apinfo -sa myprog ' to list probe names of the functions symbols in your application.

17.11 How do I reference a hardware register?

A hardware register is referenced within a user action (e.g., on_entry) by preceding the name commonly used for the register by "$$". The exact register names are documented in Appendix B, "Files Reference", under "APC File".

Note that the value you get for the register is the value it had at the point the target program called the probed routine.

On Windows, if you want the current value of a given register you can use the normal MSVC++ assembly prefixes go get it, for example:

  {     
    int CallerEAX;     
    int CurrentEAX;

    CallerEAX = $$EAX;  // move the caller's EAX to CallerEAX

    __asm mov CurrentEAX,EAX   // move EAX to the variable CurrentEAX

  }

17.12 How do I query the parameters to a function?

If the function is compiled with debug (see Q12.19 ) you can reference a parameter by name ($param) and reference all parameters with "$*.

Whether or not a function is compiled with debug,or there's an object module available, you can reference the first parameter with "$1", the second with "$2", etc., up to $8.

Note, however, that if there is no debug information provided, you must cast the "$1" to its proper type.

17.13 Can I use automatic formatting if I don't have an executable with debug information?

Yes, but you must (a) include the definition of each logged item's type in the APC file (if it's not a predefined type), and (b) cast each item to that type. This is how one can log parameters to system routines, for example:

#include <stdio.h> // includes the struct FILE 
probe thread 
{    
  probe "fopen"    
  {       
    /* fopen returns *FILE, defined in stdio.h */       
    on_exit
      log("fopen() returns ", (FILE *)$return, " = ", *(FILE*)$return);    
  }    
  probe "fclose"    
  {       
    /* first parameter to fclose is *FILE */       
    on_entry
      log("fclose() called with ", (FILE *)$1, " = ",
        *(FILE*)$1 );    
  } 
}

17.14 How do I change the return value from a function?

     on_exit { $return = desired_value; }

17.15 How do I log the value of a string parameter?

ap_StringValue is a macro which logs everything from the address provided up to the first null character:

     on_entry { log("NameParam = ", ap_StringValue($NameParam)); } 

Note : this only applies to null-terminated (C, C++) strings. It does not apply to the Ada predefined string type -- see Q17.27 .

17.16 How do I log the contents of an array?

You must specify the bounds of the array in the log statement:

     on_entry { log("Items = ", $Items[0 ..9]); }

If the array bounds are dynamic (as most are), you can compute them first

      on_entry      
    {
      int last;
      for (last = 0; $Items[last] != 0; last++);
      log ("Items = ", $Items[0 .. last-1]);      
    }

17.17 Solaris: I get a compile error when I write "a[0..4]", but it seems to work; why?

The Sun Workshop (Forte) compiler's preprocessor is run over the APC file before the APC-specific syntax is processed and converted to C. If you're using Sun WorkShop as apc's C compiler (as defined in $APROBE/lib/compiler_profile), that preprocessor complains about the "0.." syntax that APC uses. If you want to avoid the message, put a blank before ".." whenever you use it.

17.18 How do I "stub out" the probed function so it does nothing?

Use the "ap_StubRoutine" macro in the on_entry part of a function, and be sure to return something sensible if necessary in the on_exit part, e.g.,

   probe "foo" {       
    on_entry ap_StubRoutine;       
    on_exit  $return = 0;    
  }

Note that you can't assign the return value in the on_entry part, since the return register is reset as part of the stub implementation.

17.19 How do I query the data in a class from when probing a member function?

All data in a class is defined as a field of the local variable "this", so to get at the class data item "NCalls" you would do:

  log("$this->NCalls");

17.20 How do I query a global (or static) variable when there's a local one of the same name?

To specify you want a data item other than that visible by default, add an expression context string, to the target expression:

  log("static NItems = ", $(NItems, "-file items.c"));

To get the global one, if any:

  log("global NItems = ", $(NItems, "-module foo.exe"));

17.21 Can I reference a static variable that wouldn't normally be visible to my probed function?

Yes. See the previous Q. You can reference a static item by name in any file:

        log("static NItems = ", $(NItems, "-file items.c"));

even if the probed function this appears in is not in file "items.c".

17.22 Can I call a function in my program from within a probe?

If your program is compiled with debugging enabled, you can precede its name with a `$'. This is often useful for using a probe to call debugging-support routines, e.g.,

probe thread    
{       
  probe "ReadSymbolTable"       
  {          
    on_exit          
      $DumpSymbolTable($0);          
  }    
}

In the absence of debug information, you can get the symbol address from Aprobe and cast that to the correct type. For example, on Windows:

typedef void MyBeep(int Msec, int Hz);

probe thread 
{    
  probe "main"    
  {       
    on_entry       
    {          
      (*(MyBeep *)ap_SymbolToAddress               
        (ap_SymbolNameToId                  
          (ap_ModuleNameToId("KERNEL32"),                   
          "Beep()",                   
          NULL)))(4000, 1000);       
    }    
  } 
}

Additionally, on Windows, if the function is in a DLL you can use the NT routines GetProcAddress and GetModuleHandle to find the routine. For example:

  /* call Beep to sound a 2kHz tone for a second */    
  GetProcAddress(GetModuleHandle("KERNEL32"),"Beep")(2000, 1000); 

In the above example you could use the name of your DLL instead of KERNEL32.

Calling C++ methods is more complex (they require a "this" pointer, and the naming can be tricky): For Windows, see Q17.23; for Unix, see Q17.71.

17.23 Windows: Can I call a Visual C++ method from a probe?

To call Windows Visual C++ methods in the target program you will have to create a C wrapper function and link that with the application. The wrapper will take the C++ object as an explicit parameter and make the method call. For example, examine the following class definition:

class MyClass
{
public:
   MyClass () {a = 10;}

   void Show (int b);
   void Show (char *b);

private:
   int a;
};

If you want to call the first Show method (the one with an int parameter) you need to write the following wrapper function:

extern void WrapperForMyClassShowInt(MyClass *o, int b)
{
   o->Show(b);
}

You should pick a unique name for the wrapper, particularly if you are writing a number of wrappers for overloaded methods in the same class. This wrapper name includes the class, MyClass, the method name, Show, and an indication of argument types, Int.

The wrapper function includes an explicit parameter for the C++ object pointer of class MyClass, and all the other parameters for the method call, in this case just an int. The body of the wrapper function just make a C++ method call using the C++ object and the method parameters. If this method returns a value the wrapper function should just return that result.

With this wrapper function compiled and linked with your application, you can call it from APC code. Here is an example of calling the wrapper defined above:

   probe "main"
   {
   {
      on_line (31)
      {
         $WrapperForMyClassShowInt(&$m, 20);
      }
   }

The target expression $WrapperForMyClassShowInt indicates the wrapper function we wrote above, and the target expression &$m refers to a C++ object variable in the target program. Here is the function, main, that this probe targets:int main (int argc, char **argv)

{
MyClass m;

return 0; /* line 31 */
}

17.24 Can my APC files reference names in one another like a C program?

Yes, but if they do they must all be compiled in the same "apc" command into a single UAL.

17.25 Can I call a function in another UAL?

Yes. A UAL is just a shared object library (a DLL), so you must do the following:

Export the symbol for the function to be called, using the apc "-e" option, when you build the UAL to be referenced, e.g.,

   apc funcdef.apc -e func

On Windows, you can use the MSVC++ __declspec(dllexport) prefix to identify the routine as something to be exported from a DLL.

2) specify the referenced UAL (on Windows, the corresponding "lib" file) as an input file on the command-line when you compile the probe that contains the external reference flag when you specify the other UAL as a shared module

   Windows: apc main.apc funcdef.lib

   Unix: apc main.apc funcdef.ual

17.26 How do I change the return code from my Unix program?

From $APROBE/examples/learn/probe_exit/exit.apc:

probe thread {    
  probe "exit" in "libc.so" // "libc.a(shr.o)" on AIX
  {
    on_entry {          
      /* return 0 even if an error occurred: */          
      $1 = 0;       
    }    
  } 
}

Note: This probe won't work on Solaris 5.5.1 because exit() works differently.

17.27 How do I print or change a GNAT Ada string value in my probe?

An unconstrained string is represented as a record with two components. The first is a pointer to the string (which is not null-terminated) and the second is a pointer to another record which contains the bounds of the string.

The "apc" tool recognizes this special type and displays it appropriately, if debug information is available. Since it's length is known, ap_StringValue is not used. For example:

probe thread {
  probe "hello.qualify_name" {
     on_entry
     {
        // log the input parameter then stub the routine itself 
        log("qualify_name called with: ", $1
);
        ap_StubRoutine;
     }
   }
}

In the absence of debug information (e.g., for Ada.Text_IO.Put_Line ), or when you want to assign to an unconstrained string, you can use macros defined in gnatstrings.h. For example:

#include "gnatstrings.h"

probe thread {
  probe "hello.qualify_name" {
     on_exit
     {
        // return what we want to:
         ap_SetGnatUCString
(
            $return,
            ap_CatenateStrings(
               "/home/ocs/",
               ap_ExtractGnatUCString
($1),
               NULL));
     }
  }
}

17.28 How can I just log some data and format it as hex?

This is an example of an APC file to log a buffer's worth of data and format it as hex.

// Example APC file to demonstrate logging a block of data and
// formatting it as hex.

// Use this macro to provide a buffer and length of data you wish to log
// and be formatted as hex. e.g. LogAsHex (MyBuffer, 100);
#define LogAsHex(B,L)                              \
log (((ap_Byte *) ((ap_Byte *) B)) [0 .. ((L)-1)], \
     (ap_Uint32) (L),                              \
     (ap_Uint32) (B)) with HexFormat

// Buffer is the actual data, Length the length and StartAddress the
// address of the data at runtime.
static void HexFormat (ap_Byte    *Buffer,
                       ap_Uint32  *Length,
                       ap_Uint32  *StartAddress)
{
   ap_Uint32 PrintAddress;
   ap_Uint32 EndAddress;

   // We start printing at the first 16 byte boundary below StartAddress
   // which might be below where we actually need to show characters. So
   // we check if we are in range before printing a character
   PrintAddress = *StartAddress & 0xfffffff0;
   EndAddress = *StartAddress + *Length;
   
   while (PrintAddress < EndAddress)
   {
      int i;

      // Print out the hex bytes
      printf ("%08x: ", PrintAddress);
      for (i = 0; i < 16; i++)
      {
         // Check we're in range
         if ((PrintAddress + i) < *StartAddress ||
             (PrintAddress + i) >= EndAddress)
         {
            printf ("  ");
         }
         else
         {
            printf ("%02x", Buffer [PrintAddress - *StartAddress + i]);
         }

         if (i && i % 4 == 0)
         {
            printf ("  ");
         }
      }
      
      // Print out the ascii
      printf ("   ");
      for (i = 0; i < 16; i++)
      {
         // Check it's in range
         if ((PrintAddress + i) < *StartAddress ||
             (PrintAddress + i) >= EndAddress)
         {
            printf (" ");
         }
         else
         {
            ap_Byte c = Buffer [PrintAddress - *StartAddress + i];

            // Is this a printable character?
            if (c >= 32 && c <= 127)
            {
               printf ("%c", c);
            }
            else
            {
               printf (".");
            }
         }
      }

      printf ("\n");
      PrintAddress += 16;
   }
}

// This is an example of using the above log mechanism - the first
// parameter must be an address (e.g. an array, a pointer, etc.). The 2nd
// parameter is the number of bytes.
probe thread
{
   probe "fred()"
   {
      on_entry LogAsHex ($1, $2);
   }
}

A C file follows to test it with:

void fred (const char *Buffer, int Length)
{
   ;
}

int main (int argc, char *argv)
{
   char Buffer [100];
   int  i;

   for (i = 0; i < 100; i++)
   {
      Buffer [i] = (char) i;
   }
   fred ((const char *) Buffer, 100);
   return 0;
}

17.29 How do I log information about each thread as it starts?

You log the Thread ID using a format routine that prints information about it, since the information, especially the thread entry point, may not be available on_entry to the thread:

void PrintThreadInfo(ap_ThreadIdT *ThreadIdPtr)
{
  printf("Thread %d: ", *ThreadIdPtr);
  ap_PrintSymbol(
     ap_AddressToSymbol(
        ap_ThreadEntryPoint(*ThreadIdPtr)));
}                       

probe thread 
{
  on_entry
  {
     log(ap_ThreadId()) with PrintThreadInfo;
  }
}

Note that the thread entry point symbol will probably be a system function.

17.30 GNAT turns SIGSEGV into CONSTRAINT_ERROR; can I use Aprobe to get a core dump?

Yes. Here's a probe which stubs (disables) the call the GNAT runtime makes to sigaction() to register a signal handler. This allows the default action to occur when the signal occurs.

#include <signal.h>

probe thread
{
   probe "sigaction()" in "libthread.so"
   {
      ap_BooleanT Stubbed = FALSE;
      
      on_entry
      {
         if ($1 == SIGSEGV)
         {
            printf ("Stubbing sigaction(SIGSEGV)\n");
            Stubbed = TRUE;
            ap_StubRoutine;
         }
      }
      on_exit if (Stubbed) $0 = 0;
   }
}

17.31 How can get I get Aprobe actions to happen when my program dumps core?

First, you should be running with sigsegv.ual: it will provide a traceback and exit actions in these cases. If you want to add additional exit actions, such as a predefined probe snapshot, see Q15.6, or you can copy and extend $APROBE/probes/sigsegv.apc to build your own probe.

17.32 Is there a way to find out where a signal occurs when it doesn't cause a core dump?

On Solaris there is:

probe thread {
  probe "sigaction.c":"sigacthandler()" in "libc.so"
  {
    on_entry
      log("Signal ", (int)$1, " see at:");
      ap_LogTraceback(99);
  }
}

Of course, this might not work if the condition causing the signal was due to corrupted memory or registers which Aprobe relies upon.

17.33 How can I reduce the overhead of my probes?

The most obvious way is to use #pragma nofloat in probes that don't use floating point; this eliminates the need to save/restore floating point registers. See also Aprobe Performance Considerations in Chapter 4 of the Aprobe User's Guide.

probe thread
{
  probe "your_routine"
  {
    #pragma nofloat
    // Your probes
  }
}

17.34 Can I use Solaris Aprobe on JOVIAL programs?

Yes, but there will be no "debug" information found, so you won't be able to use named target expressions (e.g., "$x", "$*") or do on_line probes. Furthermore, no type information is available for parameters, etc., like "$1".

17.35 How can a log a composite object without using debug information?

A. Declare or #include a C type that maps to the structure you want, then cast your target expression to a dereference of a pointer to this C type. For example:

typedef struct 
{
   int Field1;
   float Field2;
} MyStruct;

probe thread
{
  probe "foo"
  {
    on_entry
    {
      if (((MyStruct *) $1)->Field1 > 0)
      {
        log(*((MyStruct *) $1));
      }
    }
  }
}

or perhaps a bit cleaner is:

probe thread
{
  probe "foo"
  {
    on_entry
    {
      MyStruct *Param1 = (MyStruct *)$1;
      if (Param1->Field1 > 0)
      {
        log(*Param1);
      }
    }
  }
}

17.36 How can I cast a value to a type name from the program?

"I have part of my program without debug info, but I know the type of a parameter passed in that "no debug" part, and furthermore, I know that the type name is defined in a part that does have debug info. How can I cast an "unknown-type" parameter to the known type name?"

This is similar to the previous question, except instead of defining the type in your APC, refer to the type in your program by its name and file, wrapped in "typeof", within your probe declarative part, as follows:

probe thread
{
  probe "foo"
  {
    typedef ($(MyStruct, "-file debug_part.c")) MyStruct;
    on_entry
    {
      MyStruct *Param1 = (MyStruct *)$1;
      if (Param1->Field1 > 0)
      {
        log(*Param1);
      }
    }
  }
}

17.37 Is there a special editor or editor mode for APC?

No, but it's pretty close to C. The C mode for Emacs, Lemmy, or other editor works pretty well. Contact OC Systems if you think we should put work into this.

17.38 How do I execute a probe only if a certain data condition is met?

In Aprobe version 2, you could do something like:

    probe .outer_routine
    on entry
       if $r3 = 3 then
          probe .inner_routine
            null; -- inner_routine stuff
          end probe;
       end if;
    end probe;

Probes in Aprobe2 were executable but in Aprobe3 they are declarative. You declare a named probe, and make an explicit calls to enable or disable it. For example:

probe thread
{
  probe "outer_routine"
  {
    // Note that this probe has a name "InnerProbe"
    probe "inner_routine"
    {
      ; // Inner routine stuff
    } InnerProbe;

    // Entry to outer_routine
    on_entry
    {
      if ($param1 == 3)
      {
        // We can enable or disable the probe
        ap_EnableProbe (InnerProbe);
      }
      else
      {
        // Disable the inner probe
       ap_DisableProbe (InnerProbe);
      }
    }
  }
}

17.39 How can I interactively modify the parameters to a routine in my application?

The basic approach is simple. In the little C example "t.c" below, main() calls Test() every 5 seconds, passing to it an integer and a float. Subprogram "Test" prints these values out. In t.apc We put a probe onTest, and replace the parameters with values we retrieve from the environment. The trick is how to retrive values from the environment.

One obvious way is to prompt to stdout and read from stdin. This may work for some applications, but not many. A more general approach is to check if a user created a file "Test.cfg" in the directory where the program is run and if so we read the new values of parameters with the help of a call to fscanf(). This approach works pretty well as long as the overhead of `fopen' call on entry to "Test" is acceptable. In cases when it is not one could move this call some place else and store the new values in global APC variables.

Note that this "read-a-file" approach can be used for a wide range of program iteraction. One could simply use the presence of a file as a "switch" to enable or disable certain probes.

t.c 

void Test(float parm1, int parm2) 
{
  printf("Test(%f,%d)n", parm1, parm2);
}


main()
{
  while(1)
  {
    Test(0.0, 0);
    sleep(5);
  }
}

t.apc

#include <stdio.h>

#define CONFIG_FILE "Test.cfg"

probe thread
{
  probe "Test"
  {
    on_entry
    {
      FILE *fd = fopen(CONFIG_FILE, "r");

      if (fd != NULL)
      {
         // We have a file with new values
        float Parm1;
        int   Parm2;
        fscanf(fd, "Test(%f,%d)", &Parm1, &Parm2);

        // Now update the target parameters with new values
        $parm1 = Parm1;
        $parm2 = Parm2;
        fclose(fd);
        remove(CONFIG_FILE);
      }
    }
  }
}

17.40 I'm trying to stub a function called by my program, but APC can't seem to find it.

The Ada code looks like:

    function Plock (N : in Types.Integer_T) return Types.Integer_T;
    pragma Import (C, Plock, "plock");

Plock is some system call to lock or unlock into memory process, text or data. I get a warning message from apc stating: Function "....plock[1] not found in the modules(s) provided to apc . And also an error message from apc stating: Could not resolve function name: "......plock[1]"

plock() is a system function - it is not defined within your application. The following will work:

probe thread
{
  probe "plock()" in "libc.so"
  {
    on_entry ap_StubRoutine;
    on_exit $0 = 0;   // Or whatever return you want
  }
}

17.41 Using Solaris GNAT, I want to send a signal to the program to control my probes. But the signal seems to get lost. Why?

It turns out that GNAT blocks this signal with a call to thr_sigsetmask. The following probe can be added to your existing probes to unmask this signal. This works by intercepting the thr_sigsetmask function and, if the caller is requesting to add or set the signals, removing the SIGUSR1 from the mask they provide.

#include <signal.h>
#include <thread.h>
probe thread
{
  probe "thr_sigsetmask()" in "libthread.so"
  {
    on_entry
    {
      sigset_t *NewSigset;

      // Do we have new signals or is this just a request for info?
      NewSigset = (sigset_t *) $2;
      if (NewSigset)
      {
        // What sort?
        if ($1 == SIG_BLOCK || $1 == SIG_SETMASK)
        {
          // Remove SIGUSR1
          sigdelset (NewSigset, SIGUSR1);
        }
      }
    }
  }
}

17.42 I only want to probe malloc() if it's called by realloc(). How would I do that?

Here's one way, which also illustrates some other useful idioms.

#define MyCallerFunctionId               
   ap_SymbolToFunction(                 \
      ap_AddressToSymbol(               \
         ap_LocationAddress(            \
            ap_CallerLocation(          \
               ap_CurrentLocation))))  

#define NamedFunctionId(SYMBOL,MODULE)  \
    ap_SymbolToFunction (               \
      ap_SymbolNameToId(                \
        ap_ModuleNameToId (MODULE),     \
        SYMBOL,                         \
        ap_NoName,                      \
        ap_FunctionSymbol))

probe program
{
  int MallocCalls = 0;
  int ReallocCalls = 0;
  
  ap_FunctionIdT ReallocFunctionId = NamedFunctionId("realloc()", "libc.so");
  
  probe thread
  {
    int NestingLevel = 0;
  
    probe "malloc()" in "libc.so"
    {
      #pragma nofloat
      on_entry
      {
        ap_FunctionIdT CallerFunctionId = MyCallerFunctionId;
  
        if (! ap_FunctionIdsEqual(CallerFunctionId, ReallocFunctionId))
        {     
           MallocCalls++;
        }
      }
    }

    probe "realloc()" in "libc.so"
    {
      #pragma nofloat
      on_entry
        ReallocCalls++;
    }
  }

  on_exit // from program:
  {
    log("Heap statistics on program exit");
    log("-------------------------------");
    log("Number of calls to "malloc()"  => ", MallocCalls);
    log("Number of calls to "realloc()" => ", ReallocCalls);
  }
}

17.43 I have a GNAT Ada procedure that I'm stubbing out, but want to return a string value. The procedure has a declaration similar to the one below. What's the APC?

   procedure Read_Foo (File : in  File_Type;
                       Item : out String;
                       Size : out Integer);

For routines like this, although the Item is an out parameter, GNAT implements it as if it were an in parameter (but modifiable) since the bounds of the string must already be set. The following probe shows an example of changing this:

static const char *NewString = "Aprobe string";

probe thread
{
  probe "read_package.read_foo"
  {
    on_entry
    {
      sprintf ((char *) $item.P_ARRAY, NewString);
      ap_StubRoutine;
    }
    on_exit
    {
      $return.size = strlen (NewString);
    }
  }
}

17.44 Is there a simple probe that just traces the lines in one routine?

The following gives output similar to:

   MyPackage.MyRoutine line: 120
   MyPackage.MyRoutine line: 122

when formatted:

probe thread
{
  // Replace your name here
  probe "MyPackage.MyRoutine"
  {
    on_line (all)
    {
      log ("MyPackage.MyRoutine line: ",
      ap_StringValue (ap_LineIdToNumber (ap_CurrentLineId)));
    }
  }
}

17.45 How do I reference enumeration literals in APC?

Here is an example:

a.cpp

#include <iostream.h>
#define VALUE satu

enum TYPE { sund, mond, tues, wedn, thur, frid, satu };

int main (void)
{
  TYPE bar = satu;
  cout << "Hello Worldn";
}

a.apc

probe thread
{
  probe "main"
  {
    on_line (11)
    {
      if ($bar == $satu)
      {
        log ("Match");
      }
      else
      {
        log ("No Match");
      }
    }
  }
}

If the enumeration literals are defined in a class, you can qualify them. So for:

class a
{
  enum TYPE { sund, mond, tues, wedn, thur, frid, satu };

  private:
    TYPE bar;

  public:
    void seta(){ bar = VALUE; }
};

a test;

You could use

   if ($test.bar == ($("a::satu")))

17.46 Why does including <math.h> in my APC keep it from compiling? (I want to call the "pow()" function in my probe.)

The problem here is that "log" is an Aprobe directive and it is also defined as a function in the mathematical library. So, you need a small workaround to use any function other than 'log' from the mathematical library. Here is an example:

#undef log               /* 1. undefine definition in aprobe.h */
#include <math.h>        /* 2. process math.h */
#undef log               /* 3. remove math.h's log define (AIX) */
#define log aPl          /* 4. restore aprobe's definition */

probe thread
{
  probe "main"
  {
    on_exit
    {
      log("pow(2,3) = ", pow(2,3));
    }
  }
}

The workaround is to add the preprocessor lines numbered 1 through 4 above.

If you need to use the math.h log function in an APC file, you avoid the workarounds in steps 3 and 4 above, and use 'aPl' instead of Aprobe's log operation everywhere thereafter. That is:

#undef log               /* 1. undefine definition in aprobe.h */
#include <math.h>        /* 2. process math.h */

probe thread
{
  probe "main"
  {
    on_exit
    {
      aPl("log(2.0) = ", log(2.0));
    }
  }
}

In either case, when compiling your APC file on Unix, you must pass the linker flags "-lm" as follows:

apc xxx.apc -linker -lm

because compiling any routines from the libm.a library requires the -lm flags.

17.47 Windows: How do I probe a function in a dynamically-loaded DLL?

Same as you would a pre-loaded DLL:

Write your probe like `probe "func" in "dynamic" ... `

Compile it against the DLL like `apc myprobe.apc -x dynamic.dll'

When you run your program with myprobe.dll, and dynamic.dll isn't found at program startup, the probes are "deferred" until dynamic.dll is loaded, at which time the probes are applied.

17.48 How do I query an environment variable from with a probe?

Call getenv() , as in the following example:

#include <stdlib.h> /* defines getenv() */
ap_NameT LOG_LEVEL = NULL;

static ap_BooleanT IsSevereLogLevel() 
{
  return LOG_LEVEL && (strcmp(LOG_LEVEL, "severe") == 0);
}

probe program {
  on_entry
    LOG_LEVEL=getenv("LOG_LEVEL");  /* can set LOG_LEVEL to NULL */

  probe thread 
  {
    probe "main()" 
    {
      on_entry
        if (IsSevereLogLevel()) printf("Severe\n");
    }
  }
}

17.49 The above looks like a useful utility. How can I structure my probes so it can be shared?

Here's one way, if your "utility" is pure C and doesn't use aprobe stuff.

1. Write "loglevel.h", "loglevel.c" in the obvious way, e.g.

loglevel.h

extern ap_BooleanT InitializeLogLevel(void);
extern ap_BooleanT IsSevereLogLevel(void);

loglevel.c

#include <stdlib.h>                /* defines getenv() */
#include <aprobe.h>                /* defines ap_NameT */

static ap_NameT LOG_LEVEL = NULL;

void InitializeLogLevel(void)
{
    LOG_LEVEL = getenv("LOG_LEVEL");  /* can set LOG_LEVEL to NULL */
}

ap_BooleanT IsSevereLogLevel(void)
{
  return LOG_LEVEL && (strcmp(LOG_LEVEL, "severe") == 0);
}

1. Compile loglevel.c into loglevel.o. If you #include aprobe.h , just put $APROBE/include in your include path:

cc -c -I$APROBE/include loglevel.c

1. Write the probe:

t.apc

#include "loglevel.h"
probe program
{
  on_entry InitializeLogLevel();

  probe thread 
  {
    probe "main()" 
    {
      on_entry
        if (IsSevereLogLevel()) printf("Severe\n");
    }
  }
}

1. Compile the probe, referencing loglevel.o:

apc -g t.apc loglevel.o

17.50 Can I define functions in one APC file and call them from another APC file?

Yes. See also Q17.24 and Q17.25 . This is how our predefined probes are structured. The difference is that you must provide both UALs on the aprobe command-line. One could restructure the above example like so:

1. Define the header file:

loglevel.h

extern ap_BooleanT IsSevereLogLevel(void);

1. Write the probe:

loglevel.apc

#include <stdlib.h>                   /* defines getenv() */

static ap_NameT LOG_LEVEL = NULL; 

// the externally callable function:
ap_BooleanT IsSevereLogLevel(void) 
{
  return LOG_LEVEL && (strcmp(LOG_LEVEL, "severe") == 0);
}

// initialization of data accessed by the above function:
probe program
{
  on_entry
    LOG_LEVEL = getenv("LOG_LEVEL");  /* can set LOG_LEVEL to NULL */
}

1. Compile the probe into loglevel.ual (loglevel.dll on Windows), exporting IsSevereLogLevel :

apc -g loglevel.apc -e IsSevereLogLevel

1. Write the "client" probe:

t.apc

#include "loglevel.h"
probe thread 
{
  probe "main()" 
  {
    on_entry
      if (IsSevereLogLevel()) printf("Severe\n");
  }
}

1. Compile the client probe, referencing loglevel.ual (loglevel.lib on Windows).

apc -g t.apc loglevel.ual (Windows: 
apc -g t.apc loglevel.lib)

1. When you run an application, you need both t and loglevel:

aprobe -u t -u loglevel my_program

17.51 I am trying to write an aprobe that will call an Ada routine in a package body, but the routine never seems to get called.Why?

Presumably because the probe on that function is not triggered. That's because we disable probes whilst in an entry action. This is pretty easy to understand given an example. Suppose you have the following probe:

probe thread
{
   probe "printf()" in "libc.so"
   {
      on_entry printf ("We're in printf\n");
   }
}

Obviously if Aprobe didn't do anything specific, you would end up in an infinite loop: Your code would call printf() which would call the entry action for printf which would call printf which would call the entry action ... So what we do is disable the probes while you're in an action. That way the call to printf() from your probe wouldn't trigger the probe on printf itself.

In your example you are calling a routine while probes are disabled so the probe on that routine doesn't get triggered. Of course you can manually turn probes on yourself (although it is then your responsibility that you won't allow an infinite loop). The description of this in aprobe.h was improved in version 3.1.7, to the following:

These two routines

     extern void ap_IncrementDisableProbesCount (ap_ThreadContextPtrT);
     extern void ap_DecrementDisableProbesCount (ap_ThreadContextPtrT);

can be used to turn off / on probes for the thread. Normally when a probe is hit, Aprobe disables further probes in the thread for the duration of the action. This is to prevent recursive loops (for instance imagine if a probe on "printf()" called "printf()" and we did nothing about it). Sometimes you may want to temporarily enable probes. For instance, suppose on_entry to routine A you make a call to another routine in your application (say B) which calls routine C. You have a probe on C which you want to happen. You could bracket the call as follows:

on_entry
      {                                                       
         // Turn on probes before the call                    
         ap_DecrementDisableProbesCount (ap_ThreadContextPtr);
         // Make the call                                     
         $B (1, 2, 3);                                        
         // Turn probes back off                              
         ap_IncrementDisableProbesCount (ap_ThreadContextPtr);

So, your probe becomes:

probe thread
{
    probe "test.adb":"test.x[1]"
    {
      on entry
        ...
      on exit
        ...
        ap_DecrementDisableProbesCount (ap_ThreadContextPtr);
        $("test.y[1]");
        ap_IncrementDisableProbesCount (ap_ThreadContextPtr);  
    } 
}

17.52 How can I log a string passed to a library function like strdup() where there's no debug information?

In the absence of debug information all parameters would be assumed to be of type 'int' and only positional ($1, $2, etc.) references will be allowed.

If you know the type of such parameter you could cast it to the right type. The strdup() function doesn't have debug information, but you could still compile and use the following apc file:

probe thread
{
   probe extern:"strdup()" in "libc.so"
   {
   on_entry
      log("strdup(", ap_StringValue($1), ")");
   }
}

Note that ap_StringValue is a macro which among other things casts the argument to a string.

For a complete list of subprograms that you can probe in shared libraries do:

aprobe -u info -p -sa <your_executable_here>

It is best not to mix apc code that relies on debug information with the apc code that should compile without it. This way when you compile the apc code that doesn't require debug info you may omit the -x option altogether and you would not have any warnings from the apc compiler.

17.53 Can I use Aprobe to change the command run by a call to system() from my application to run my own little script instead?

Yes: replace the parameter to system() with a path to your script. In this example, the new path fits in the space occupied by the old. Imagine the possibilities...

my_ls.apc

// change these 2 lines to work on a different command:
static char cmd_to_change[] = "/bin/ls";
static char my_script[]     = "/tmp/my_ls ";

probe thread
{
  probe "system()" in "libc.so" // or libc.a(shr.o) for AIX
  {
    ap_NameT new_command = NULL;

    on_entry
    {
      char *command = (char *)$1;

      // for debugging, give some info about where we are:
      log("system() called with ", ap_StringValue($1));
      ap_LogTraceback(99);
      
      // make sure we only replace the right command
      {
        char *cmdpos = strstr(command, cmd_to_change);
         if (cmdpos == command)
        { // replace it
          char *argstring = command + strlen(cmd_to_change);
          new_command = ap_CatenateStrings(my_script, argstring, NULL);
          $1 = (int)new_command;
          log("*** changed to: ", ap_StringValue($1));
        }
      }
    }
    
    on_exit
      // indicate the return code for the command:
      log("system() returns ", $0);
      // free our string:
      ap_StrFree(new_command);
  }
}
my_ls script

echo "MY_LS: --->"
ls -ltF
echo "<---- MY_LS"

17.54 Is there a way to catch and suppress exceptions?

We do support suppressing C++ exceptions by calling Aprobe for AIX, but not Solaris or Windows. On AIX the syntax is:

probe "fred"
{
  on_exit
      if (ap_ProbeActionReason ==
        ap_CppExceptionPropagated)
          ap_SuppressException;
}

You can catch exceptions in the on_exit section of your probes. To catch exceptions all you have to do is to distinguish between a normal exit from your subprogram and an exception exit from it as both would trigger your probe's on_exit actions. For example, if subprogram "fred()" may leave via exception you could test for this as follows:

  probe thread
  {
    probe "fred()"
    {
      on_exit
        switch(ap_ProbeActionReason)
        {
          case ap_AdaExceptionPropagated:
          case ap_CppExceptionPropagated:
            log("Exception exit from fred()\n");
        }
    }
  }

If you need to, you can find other action reasons defined in aprobe.h.

The example above works well when you know where the exception may be raised, when you don't know you can log all exceptions raised in your program. To do so use the following probe:

probe thread
{
   ap_LogExceptionsInThread;
}

There are also other macros for this: ap_PrintExceptionsInThread , ap_PrintAndLogExceptionsInThread . These are all defined in aprobe.h

17.55 I'd like to probe routines in the Windows sockets DLLS. Any issues I should be aware of?

Here's what we've found:

1. Probe WS2_32.dll as that's where all the other DLLs forward socket requests. There are a couple of MS specific routines in mswsock.dll too.
2. Generally WS2_32.dll is loaded dynamically so it will be hard to see what sorts of functions it has available. You can use the following command line to get the symbol names from ws2_32.dll:

   aprobe -u info.dll -p -sa -dll ws2_32.dll foo.exe > out.txt

• Where foo.exe is some executable in your local directory. The -dll <filename> switch causes the dll to be loaded at the start of a program and thus its symbols are available for info.dll to print.

• "Bad things will happen if you probe the routine "WEP" in ws2_32.dll."

17.56 Can I track stack usage with Aprobe?

A probe to track stack usage is available here for Windows and AIX. The AIX should be easily extended for other Linux and Solaris.

17.57 Is there a way to access local variables that doesn't depend on a hard-coded line number?

Yes. Function-relative line numbers are supported using an expression consisting of a constant offset from the special values 'first' and 'last'. For example:

   probe "Outer"
   {
      // Assume that 30 is the relative line number for the next line
      // after the call to Inner
      on_line (first + 30)
      {
         $i = 99;
      }
   }

To be sure you're using the right value, you'll have to know the probe-able lines in your function (see Q17.66). The offset is then the difference between that line and the probe-able line you want (e.g., if the first line is 12, and you want line 22, then probe on_line (first + 10).

Now if the file changes your probe will still work unless you modify Outer (which is obviously less of a concern since that's the one your working with anyway).

17.58 Can I use Aprobe to query a caller's local data that wouldn't be visible by normal visibility rules?

What you might want to do is hold the address of the variable and then change that.

   probe thread
   {
      int *i;

      probe "Outer"
      {
         on_line (first)
         {
            // Store the address of i
            i = &$i;
         }
      }

      probe "Inner"
      {
         on_entry
         {
            // Change the value of i
            *i = 100;
         }
      }
   }

Obviously this is harder for types that aren't straight integers, etc. The typeof expression can be useful here:

   probe thread
   {
      typeof ($("myrecordt", "-file types.ads")) *RecordPtr;
   }

17.59 In APC I can reference some class members as fields of class objects, but others I cannot. Why?

Here are some general limitations and workarounds for accessing class data and methods:

1. Class static data is not part of the object; it is a global and is referenced using a qualified name, like

          $("Screen::nNumScreens")

• If you're unsure of the full name of a static data item you can use:

          apinfo -d myprog.exe

1. A class object is always called $this within a method. However, static class methods do not have a $this argument.
2. To see what's really in the class object, use "log(*$this);" on_entry to a method.
3. If you're unsure of the full method name in class "Class", you can use

          apcgen -L <dll-or-exe> | grep "Class::"

• or

          apinfo -sa myprog.exe | grep "Class::"

Here's a simple example:

////////////////////////////////////////////////////////////
// TestStatic.apc
////////////////////////////////////////////////////////////
probe program
{
  on_entry
    printf ("  p. Static1.exe execution has started\n");
  on_exit
    printf ( "  p. Static1.exe execution has completed\n");

}
probe thread
{
  probe "Screen::Screen"
  {
    on_entry
      printf ("  p. New screen has been constructed!\n");
  }
  probe "Screen::~Screen"
  {
    on_entry
      printf ("  p. A SCREEN HAS BEEN DESTRUCTED!\n");
  }
  probe "Screen::Update(void)"
  {
    on_entry
      printf ("  p. A screen update has started!\n");
      printf ("  p. Within Update, Current nNumScreens =%d\n",$Screen::nNumScreens);
  }
  probe "Screen::GetNumScreens(void)"
  {
    on_entry
      printf ("  p. GetNumScreens has started!\n");
      printf ("  p. Current nNumScreens = %d\n",$Screen::nNumScreens);
  }
  probe "main()"
  {
    on_entry
      printf ("  p. Main() has been started!\n");
  }
}

17.60 How can I enable and disable probes externally while my program runs?

You can do this by periodically checking for the existence of a file. If you find the file enable the probe. You can automatically delete it from your probe if you want a single-action check, or delete it yourself when you want to disable the action again. For example:

static ap_BooleanT MemsetProbeEnabled = FALSE;

probe thread
{
   probe extern:"memset()"
   {
   // We are not using floating point registers.
   // Use nofloat pragma to avoid saving them and
   // speed things up a little.
   #pragma nofloat
   on_entry
      if (MemsetProbeEnabled)
      {
         // Log parameters, traceback, etc.
      }
   }
}

#define CONFIG_FILE "/tmp/memset.cfg"

static void PeriodicAction(void *EP)
{
   FILE *fd = fopen(CONFIG_FILE, "r");

   if (fd != NULL)
   {
      // Togle the value of MemsetProbeEnabled
      MemsetProbeEnabled = !MemsetProbeEnabled;

      fclose(fd);
      remove(CONFIG_FILE);
   }
}

probe program

{
on_entry
   ap_DoPeriodically(
      PeriodicAction,
      15, // interval in seconds
      NULL);
}

17.61 AIX: How do I convert my pre-version-3 APC file to current one?

Aprobe version 2, which was delivered with OC Systems' PowerAda and OATS products as well as being sold separately for C and C++, was fundamentally different in its processing and expression of APC.

The best way isn't to "convert" at all, but to understand what the probes in your old APC file are trying to do, read the current documentation about Aprobe, and then write a probe to do the same thing in Aprobe version 4. This answer will just enumerate a few of the key differences, and rely on you to look in the user's guide for details:

Version 2 Aprobe was available only for the AIX platform, and used low-level AIX register and symbol names. Aprobe versions 3 and newer support multiple platforms.

In Version 2, "aprobe" actually compiled each APC file at run-time. In Version 4, you use the new `apc' program to compile the APC file(s) into a linkable UAL file, and name the UAL files on the aprobe command line.

Version 4 APC is C with a few extra keywords. Version 2 was an invented language based on Ada syntax. So, for example, instead of case $r3 is ... you'd write switch($$r3) { ...

In Version 4 there's an underscore to make "on entry" one word: "on_entry", "on_exit", "on_line".

In Version 2 you could write

probe .sym1, .sym2, on entry ...."

In Version 4 each probe can name only one symbol, but there is the new concept of a "probe type" or "typedef probe" which may be defined and then applied to many symbols. So you'd do

typedef probe { on_entry ... } CoolProbeT; 
CoolProbeT  Sym1Probe("sym1");
CoolProbeT Sym2Probe("sym2");

In Version 2 APC there were only registers ($r3). In Version 4 you can reference parameters by position ($1, $2, etc.); In Version 4 you can reference the return value on_exit as $0, and that's not to mention accessing program variables by their source names...

Because version 2 APC was so low-level, there was another tool "apgen" which read an "apg" file that supported a few operations on source-level variables and generated APC to access them. In Version 4, you can reference a source-level name anywhere, provided that name is available in the debug information of the executable provided to the apc compiler.

In Version 2 a `format' was required for each log statement, and was a special syntax that could be named or unnamed in-line. In version 4 a format routine is just a C routine which can be automatically generated based on the types of the `log' arguments.

Here is a reply given to a customer who asked this question:

> It is my understanding that the new aprobe is more "C like" than "ADA like".
>Beyond that, I could use a little help.

That's true - it is basically ANSI C with some extra keywords. I take it you have gone through the examples in $APROBE/examples/evaluate to get yourself acquainted with the syntax? If not do that first and then come back to your larger problem.

> I wasn't sure if the [Aprobe v2] words format and bytes were aprobe terms.

Yes they are. In v2 you had `format start' and `format finish': These have been made consistent with all other probes on v4 so you would use:

probe format
{
  on_entry
  {
    // Put the equivalent actions to the format start here
  }

  on_exit
  {
    // Put the equivalent actions to the format finish here
  }
}

The bytes operator was a v2 thing. In v4 you would express the code in
terms of C so you would probably use char [] :

  on_entry
   {
    char CmdText [200];
  }

> I wasn't sure about the $function.

This is where v4 is much better than v2. Since you are writing your probes in C, you can just include the header files and call the functions directly. For instance, you wish to call the `creat' function. All you need to do is:

// Include the header files
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

probe format
{
  on_entry
   {
    int fd;
    fd = creat ("Filename", 0644);
  }
}

and the same for access, system, printf, sprintf, write, etc. You'll find your probes look much better!

> The probe part I did I am pretty sure is wrong.

`probe' is quite different and you have to account for the different names used by GNAT and the registers used on SPARC [compared to PowerAda and AIX]. Here's an example which will be close:

probe thread {
   // I'm guessing on the name here: If you have trouble finding the
   // routine, run `aprobe -u info.ual -p -s <exe name> > syms' and all of
   // the routines will be placed in the syms file.
   probe "Queuing_Services.Read_From_Q[2]"
   {
      // Store the parameters on entry since the registers aren't
      // available on_exit
      int   SrNum = $2;            // Second parameter was $r4 on AIX
      int   Length = $3;           // Third parameter was $r5 on AIX
      char *Data = (char *) $4;    // Fourth parameter was $r6 on AIX

      on_exit
      {
         // Log the data
         log (SrNum, Length, Data [0 .. Length - 1]) with DitFormat;
      }
   }
}

Your format routine should be defined above this; in v4 they are regular C routines but the important thing is that they take pointers to the data, so:

void DitFormat (int *SrNum, int *Length, char *Data)
{
   // Do your processing here
}
      

A couple of comments on the new file: It is recommended that you use C++ style comments (//) unless you wish to keep code common with some existing C code since they are less error prone.

Make sure your format routine only has pointers for it's parameters.

Hope this helps - like I said, make sure you understand how to write simple probes, logs and logs with formats and then you should be fine to tackle this exercise.

17.62 (Unix) Is there a probe to see when my application "exec's" another program?

Here is the source for a probe that should do the trick. It will record calls to all of the exec routines, including the file, calling user/group IDs, file user, group and mode information and the environment. It was written for Solaris but should work on other Unixes.

To compile, just save to your local disk and do apc faq_exec.apc.

To use this probe you will need to have a new or existing workspace for the process you want to watch. Then either,

• Copy the exec.ual file into the workspace directory, or
• Use the Add Ual option from the Setup menu in the main RC window. In the "Ual file" field type or browse to the exec.ual file. Uncheck the copy UAL checkbox and (optionally) give it a description like "Record exec calls" and click OK. The UAL will be listed in the list on the left hand side of the window; check the checkbox for it and click Build.

Although the first option is simpler, using Add Ual will make it easier to turn on or off later.

Now rerun, format and look for the exec calls. If necessary the probe can be expanded to record parameters if this will be necessary to identify it.

17.63 How can I cast an enumeration value to print its numeric value?

Yeah, the "obvious" direct cast doesn't work. The trick is to get that byte into something you can safely cast. The reliable way to do it is as shown below.

probe thread {
  probe "qts.write_to_q" {
    on_exit
    {
       /* this doesn't work: log("rc=", $rc, "=", (int)($rc)) */
       char rc_val = *(char *)&$rc;
       log("rc=", $rc, "=",  (int)rc_val);
    }
  }
}

17.64 Is there a probe that will print a static call tree of my executable?

We did a rough one which just lists the calls made by each function in a Solaris/Sparc executable. You can grab this here.

It works by disassembling the object module and recognizing the call instructions, so it would take some work to port to other platforms. If you're really interested in having it extended or ported, please contact us.

17.65 How can I detect memory overwrites on dynamically allocated (malloc'd) memory?

A crash can happen because memory allocated using malloc() or its variants is being corrupted by code that writes past the end (or before the beginning) of the memory that's returned, corrupting malloc's internal pointers or adjacent data.

The predefined "memcheck" probe detects this by putting a "fence" at the end of allocated memory, and checking the fence is intact when the memory is freed: see Q15.18.

17.66 (Unix) How do I know when my application has forked?

You can use the ap_AddNewProcessCallback to add a callback when Aprobe detects your new process. Pass it a handler that will be called in the child process. For instance:

static void MyNewProcessHandler (ap_ThreadContextPtrT ThreadContext)
{
   log ("Here is my new process");
}

17.67 How do I know what lines I can probe in a function?

The most reliable way is to use:
   apcgen -qlines -p function_name -x module_name

This generates an on_line section for each line in the given function. You can redirect the output to a file and edit the file with your on_line actions.

For an executable module you can use:
   apinfo -l exe_name
which lists all the symbols and their lines, if any. This output is simply the "raw" line information, sorted by code offset, so is not as useful for writing probes, though the output may be a good reference for use with test coverage or a debugger.

17.68 (Windows) How can I track page faults using Aprobe/RootCause?

In general, explaining why a Page Fault occurred is difficult. Note that there's a lot more Paging information available then just a fault count (e.g. current and peak Virtual memory, current and peak real memory, current and peak page file usage, etc.). The following is offered in an attempt to answer this question.

For process-specific page fault, or general memory usage information, you can use the GetProcessMemoryInfo() function, which is documented in MSDN. Here's the data structure returned:

typedef struct _PROCESS_MEMORY_COUNTERS {
    DWORD cb;
    DWORD PageFaultCount;
    DWORD PeakWorkingSetSize;
    DWORD WorkingSetSize;
    DWORD QuotaPeakPagedPoolUsage;
    DWORD QuotaPagedPoolUsage;
    DWORD QuotaPeakNonPagedPoolUsage;
    DWORD QuotaNonPagedPoolUsage;
    DWORD PagefileUsage;
    DWORD PeakPagefileUsage;
} PROCESS_MEMORY_COUNTERS;
typedef PROCESS_MEMORY_COUNTERS *PPROCESS_MEMORY_COUNTERS;

You can also use the native API ZwQueryInformationProcess() which has the capability of returning more information than GetProcessMemoryInfo() (which uses this native API).

For System-Wide page fault, or general memory usage information, you can use the native API ZwQuerySystemInformation() to get all sorts of performance data.

For both above cases, you can either create a thread that calls the function periodically to collect sampling data, or better is to use the Aprobe ap_DoPeriodically() function, which does this for you.

You could integrate this (or other system-wide statistics) with tracing in a manner similar to the RootCause "perf_cpu" probe. Contact us if you'd like some help with this.

Also note that you can use the Windows "PerfMon" feature to generate real-time graphs of these statistics correlated with program points. See Perfmon in the Aprobe User's guide.

(Note that the "Zw" native APIs mentioned above are not "officially" documented by Microsoft, but they are widely used in both user and Device Driver development are are safe to use. They are documented in "Windows NT/2000 Native API Reference" by Gary Nebbett and you can find them in DDK and web documentation as well).

17.69 Is there a routine available to find symbol ids by mangled name, or one that will demangle for us?

You can generally pass a mangled name as the name to ap_NameToSymbolId() and you'll get the correct Symbol ID. However, there is also the following (defined in aprobe.h, of course):

extern void ap_Demangle(
   ap_DemangledNameT *Result,
   ap_NameT          MangledName,
   ap_BooleanT       IsSubprogram,
   ap_CompilerKindT  CompilerKind);

Here is an example of how to use it:

{
   ap_DemangledNameT DemangledName;

   ap_Demangle(
      &DemangledName,
      ".sec_fdk_Nam_Svc_Def__ELAB",
      TRUE,
      ap_AIXpa4_CompilerKind);


   // Now we can use DemangledName.FullName
   SymbolId =
      ap_SymbolNameToId(
         ap_ApplicationModuleId(),
         DemangledName.FullName,
         ap_ExternSymbol,
         ap_FunctionSymbol);
}

17.70 Is there a way to suppress (or force) the warning when probing a symbol that is undefined?

Yes, this was introduced in RootCause 2.1.3/Aprobe 4.3.3 (February 2004). The way to do it is specify #pragma optional in column 1 immediately inside the probe (or typedef probe), for example:

probe thread {
   probe extern:"PrintDebug()" {
#pragma optional
    ...
   }
}

Conversely there is also a #pragma required which forces a warning in the case where the module is undefined. By default, a warning is not generated on probes on missing modules. For example:

probe thread {
   probe extern:"open()" in "libpthread.so" {
#pragma required
   }
}

This was possible prior to version 2.1.3 but was harder since it required use of a typedef probe and programmatic checking and instrumentation using the Aprobe API. (See for example AllocationFunctions[] array in memwatch.apc.)

17.71 Unix: Can I call a C++ method from a probe?

Yes, if:

• You know the full method name, and
• You have a this pointer for that method's class available (or else the method is static).

In these cases, you call it just like a C function (see Q17.22 except that you pass this as the first parameter). For example, suppose you have a class that looks something like this:

class Example {
public:
   void doIt(const string& s);
   void debugIt(const string& s);
};

And you want to call debugit() on entry to doit(). The following works:

probe thread {
  probe "Example::doIt" {
    on_entry {
      $("Example::debugIt")($this, &($s));
    }
  }

But obviously this is a very simple example. In many real cases you have template instances with long and subtly different names. In such cases, you can use apcgen -vL to list the methods in an individual object file and "grep" for the methods you're looking for and try to match up the line numbers.

When you have dynamically dispatched calls, you are limited to methods in common base classes, or else you need to use some conditional test to determine which specific method to call.

Often the best choice is to use a separate extern "C" C++ module as an interface between your probe and the call, as described in Q17.23.

As always, if you have problems or questions, contact .

// example.cpp - setting and getting std::string values
// compile with '$(CCC) -g example.cpp -o example.exe'
//   where '$(CCC) is xlC for AIX, CC for SunWorkshop, g++ for GCC
// run with zero or more arguments, e.g., 
// $ example.exe one
// will print
// Example="example.exe"
// Example="one"

#include <iostream>
#include <string>

using namespace std;

static void print_string(string &s)
{
   cout << s;
}

class Example
{
public:
  void put_string(char *val);
  string get_string(void);
  void print_string(void);
private:
  string value;
};

string Example::get_string(void)
{
   return value; 
}

void Example::print_string(void)
{
   cout << "Example=\"";
   ::print_string(value);
   cout << "\"" << endl; 
}

void Example::put_string(char *val) 
{ 
   value = val;
}

int main(int argc, char **argv)
{
   string Str;
   Example X;
   for (int i = 0; i < argc; i++)
   {
      X.put_string(argv[i]);
      Str = X.get_string();
      // ::print_string(Str);
      X.print_string();
   }
   return 0;
}

// cppstring_help.cpp - C++ functions supporting cppstring.apc

#include 
extern "C"  {

const char *get_string(void *std_string_ptr)
{
   std::string *s_ptr = (std::string *)std_string_ptr;
   return s_ptr->c_str();
}

void set_string(void *std_string_ptr, const char *from)
{
   std::string *s_ptr = (std::string *)std_string_ptr;
   *s_ptr = from;
}

}

// cppstring_help.h - C++ functions supporting cppstring.apc

extern const char *get_string(void *std_string_ptr);
extern void set_string(void *std_string_ptr, const char *from);

#define GET_CPP_STR(S)  get_string((void *)&S)
#define SET_CPP_STR(S, NEW_CS)  set_string((void *)&S, NEW_CS)

// cppstring.apc - setting and getting std::string values and parameters 
//                  from a probe.  
// OTHER REQUIRED FILES: cppstring_help.h, cppstring_help.cpp
// BUILDING THE UAL:
// All:
//   $(CCC) -c cppstring_help.cpp # create cppstring_help.o for your platform
// Solaris:
//   apc cppstring.apc -x example.exe -u cppstring_help.o 
// AIX:
//   apc cppstring.apc -x example.exe cppstring_help.o -linker "-lC"
// Linux:
//   apc cppstring.apc -x example.exe cppstring_help.o -linker "/usr/lib/libstdc++.so.6"
// Windows:
//   apc cppstring.apc -x example.exe cppstring_help.obj
// 
// RUNNING
// Run with 'aprobe -u cppstring example.exe' to get output like:
// put_string: Changing example.exe to probe_string1
// get_string: Changing probe_string1 to probe_string2
// print_string: Changing probe_string1 to probe_string3

// this defines the macros GET_CPP_STR and SET_CPP_STR
#include "cppstring_help.h"

// The replacement strings:
static char probe_string1[] = "probe_string1";
static char probe_string2[] = "probe_string2";
static char probe_string3[] = "probe_string3";

probe thread
{

   probe extern:"Example::put_string(char*)"
   {
      on_entry
      {  // change entry parameter to be a new string 
	 printf("put_string: Changing %s to %s\n", $1, probe_string1);
	 $1 = probe_string1;
      }
   }

   probe extern:"Example::print_string(void)"
   {
      on_entry
      {  // change entry parameter to be a new string 
	 printf("print_string: Changing %s to %s\n", 
	     GET_CPP_STR($this->value),
	     probe_string3);
         SET_CPP_STR($this->value, probe_string3);
      }
   }

   probe extern:"Example::get_string(void)"
   {
      on_exit
      {  // change return value parameter to be a new string 
	 printf("get_string: Changing %s to %s\n",
	     GET_CPP_STR($return),
	     probe_string2);
	 SET_CPP_STR($return, probe_string2);
      }
   }
}

18. Writing Java Probes

// The application Simple.java
public class Simple
{
   int doIt ()
   {
      return 10;
   }

   public static void main (String[] args)
   {
      System.out.println ("doIt returns " + new Simple ().doIt ());
   }
}

// The probe SimpleProbe.java
public class SimpleProbe extends com.ocsystems.aprobe.ProbeMethod
{
   public Object onExit (Object returnValue)
   {
      return new Integer (11);
   }
}

<!-- The xmj file simple.xmj -->
<probe_deployment>
   <probe class="SimpleProbe" parameters="readonly">
      <target value="Simple::doIt"/>
   </probe>
</probe_deployment>


$ javac Simple.java
$ javac -classpath $APROBE/lib/aprobe.jar SimpleProbe.java
$ apjava -u simple.xmj -java Simple
doIt returns 11

import com.ocsystems.aprobe.*;
public class TestProbe1 extends ProbeMethod
{
   public boolean onEntry(Object[] parameters)
   {
      stub();
      return true;
   }
}

...
  SquareID id = (SquareID) params[0];

  id.otherMethod();
  return true;
...

   probe "a()" {
     probe "b()" {
       on_entry
          do_something();
     }
   }

19. Logging Data

        log((int) x) with myformat;

        static void myformat(int *i) { ... };

int foo[10];

log(foo[0]) with MyFormat;
log(foo[0..9]) with MyFormat;

log(foo[0..0]) with ... // Do you use 'int *' here or 'int'
?  log(foo[Var1..Var2] with ...  // We don't even know the number of
elements here.  

log("foo[0] => ", foo[0]);

20. Other Aprobe Questions

      clock_gettime( CLOCK_REALTIME, ap_TimeT_ptr);

   if (ap_CurrentAprobeState() == ap_AprobeFormatTime) { ... } 

  probe program {       
    on_entry {          
      DumpInfo();          
      // Don't run the program.  Exit after printing all the info.
      // (MAGIC exit code tells runtime this is *not* and error)
      exit(APROBE_MAGIC_EXIT_CODE); 
    }
  }

  probe format {      
    on_entry {          
      if (ap_CurrentAprobeState () == ap_AprobeFormatTime)
      {             
        DumpInfo();             
        /* Don't do any formatting.  Exit after printing all the info. */
        exit(0);          
      }       
    }    
  }

#define ap_StructValueReturnExpected private

probe "UpdateCoordinates()"
on_exit
  if (ap_StructValueReturnExpected) 
    $return.x = $return.y = $return.z = 0;
}

typedef struct
{
  int a;
  int b;}
MyStructT;

void foo(MyStructT &MyS) {}

main(){
  MyStructT S = {10, 1999};
  foo(S);}

probe thread
{
  probe "foo"
  {
    typeof($1) *Param1 = &($1);
    on_exit
    {
      log("Param1 => ", *Param1);
    }
  }
}

probe thread{
  // Method 1: Use the APC "typeof" operator on the type name directly as a
  //           target expression:
  probe "foo"  {
    typeof($MyStructT) *Param1 = &($MyS);
      on_exit {
      log("Param1 => ", *Param1);
    }
  }

  // Method #2: Use the APC "typeof" operator on the target
  //            expression for the parameter name:
  probe "foo"{
    typeof($MyS) *Param1 = &($MyS);
    on_exit{
      log("Param1 => ", *Param1);
    }
  }

  // Method # 3: Use the "typeof" operator on the target expression
  //             for the positional parameter.
  probe "foo"
  {
    typeof($1) *Param1 = &($1);
    on_exit
    {
      log("Param1 => ", *Param1);
    }
  }
}

  aprobe -u info -p -s myprog.exe | grep Method

or

  apcgen -Lv myprog.exe | grep Method

[Enter : extern:"dyndll2d1_a()?0" in "dyndll2d1.dll" at 10:08:02.799519095
 [Enter : extern:"dyndll2d1_a()" in "dyndll2d1.dll" at 10:08:02.799526359
  [Enter : extern:"dyndll2d2_c()" in "dyndll2d1.dll" at 10:08:02.799534181
  ]Leave : extern:"dyndll2d2_c()" in "dyndll2d1.dll" at 10:08:02.800248518
 ]Leave : extern:"dyndll2d1_a()" in "dyndll2d1.dll" at 10:08:02.800256899
]Leave : extern:"dyndll2d1_a()?0" in "dyndll2d1.dll" at 10:08:02.800262486 

probe thread
{
    probe "myfunc()"
   {
       ap_BooleanT IsEnabled = TRUE;
   on_entry
       if (some_expression)
       {
          IsEnabled = FALSE;
          return;
       }
    on_exit
      if (! IsEnabled) return;
    on_entry
       do_something();
    on_exit
       do_something();
   }
}

$ aprobe -u t1 -u t2 t.exe
enter t2:main()
enter t1:main()
exit t1:main()
exit t2:main()

apc MyClassProbe.apc CallMyClass.o -u -x MyProgram

aprobe -u MyClassProbe.ual MyProgram

#include <signal.h>
#include "memwatch.h"

static void Handler (int sig, siginfo_t *siginfo, void *ucp)
{
   printf ("Taking snapshot on signal %d\n", sig);
   ap_Memwatch_DoSnapshot ("snapshot signal");
}

probe program
{
   on_entry
   {
      ap_RegisterSignalHandler (SIGPROF,
                                ap_CallBeforeUserAction,
                                Handler);
   }
}

apc memwatch_sig.apc memwatch.ual

  log ($available_overlays [1] [1]);

  log ($available_overlays [1] [1 .. 10]);

// Define a type to map to the routine
typedef int (*pthread_attr_getstacksize_subprogram_T)
   (pthread_attr_t *, size_t *);

// Declare a variable to hold the address
static pthread_attr_getstacksize_subprogram_T
   pthread_attr_getstacksize_subprogram_ptr = NULL;

probe program
{
   on_entry
   {
      pthread_attr_getstacksize_subprogram_ptr =
         (pthread_attr_getstacksize_subprogram_T)
            ap_FunctionPointer (ap_ModuleNameToId (PthreadModuleId (),
                               "pthread_attr_getstacksize()",
                               ap_NoName);

      // Call it - note don't do this on program entry until you have the
      // fix for that!
      if (pthread_attr_getstacksize_subprogram_ptr)
      {
         pthread_attr_getstacksize_subprogram_ptr (&Attributes, &Size);
      }
   }

The PthreadModuleId () routine would look something like:

static ap_ModuleIdT PthreadModuleId(void)
{
   ap_ModuleIdT Result;
   /* First, the 4.3 case */
   Result = ap_ModuleNameToId("libpthreads.a(shr_xpg5.o)");

   if (ap_IsNoModuleId(Result))
   {
      /* Didn't find it in shr_xpg5.o, so if we don't find it in shr.o ...
*/
      Result = ap_ModuleNameToId("libpthreads.a(shr.o)");
      /* ...we'll give back that null result. */
   }
   return Result;
}

int *GetThreadSpecificInt();

probe thread
{
   int ThreadSpecificItem = 0;

   int *GetThreadSpecificInt()
   {
      return &ThreadSpecificItem;
   }
}

probe thread
{
   on_entry
     printf("Entering thread\n");
   on_exit
     printf("Exiting thread\n");
}

   LDR_CNTRL=MAXDATA=0x20000000 c2.eab
or
   LDR_CNTRL=MAXDATA=0x20000000 apformat c2.apd

  56.7    0.59     Other functions (not in profiled module)

#include <sys/types.h>
#include <unistd.h>
probe thread {
 probe "somewhere_where_there_can_be_a_problem" {
   on_line (where_there_can_be_a_problem) { // or on_entry or on_exit
     if (the elusive problem the user is watching for has occurred) {
// here is the guts of the probe
       int normal = $some_reference; // save a normal state, explained below
       pid_t child;
       child = fork();
       if (child) fprintf(stderr,
          "Oops -- such-and-such happened -- gdb xxx %d\n", child);
       else while (++child) {
         if (child > 600) exit(1); // kill if unused in ten minutes
         if ($some_reference==normal) sleep(1); // stay in the probe
         else {$some_reference = normal; break;} // leave the probe
       }
     }
   }
 }
}

probe thread
{
   probe extern:"::myroutine(void)"
   {
      on_entry
      {
         // The list is in a variable called my_list.
         // We need to call list.size ():
         log 
($("list<basic_string<char,string_char_traits<char>,__default_alloc_template<false,0> >,allocator<basic_string<char,string_char_traits<char>,__default_alloc_template<false,0> > > >::size(void)const") (&$my_list));
      }
   }
}

21. Licensing

OC Systems spends a surprising amount of time helping users get licensing set up on their machines. Here are a few of the most common questions and answers.

SERVER my.server.name 000347b371fe

SERVER my.server.name 000347b371fe 27001

mkitab -i rcnfs rclocal:2:once:/etc/rc.local