OCS Release Notes Aprobe 4.5.4ai Linux32

From OC Systems Wiki!
Revision as of 17:02, 22 September 2021 by Swn (talk | contribs) (Created page with "<pre> *READ THIS FIRST* Release Notes for INFORMAL RootCause Version 2.3.4ai (Aprobe Version 4.5.4ai) for AIX...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
                          *READ THIS FIRST*

                          Release Notes for
        INFORMAL RootCause Version 2.3.4ai (Aprobe Version 4.5.4ai) for AIX
                            Sep 27, 2021

This file contains brief release notes for RootCause 2.3.4ai and Aprobe
4.5.4ai for 32-bit AIX.  Changes are relative to RootCause version
2.3.4 (Aprobe 4.5.4) released Sep 2021.

Features and Fixes in RootCause 2.3.4ai (AIX32 platform)
 (2721) statprof -r option produces odd resutls

Features in prior releases are listed below (look for "PRIOR").

Getting Help

If you have any questions or problems, please contact OC Systems by
e-mail at support@ocsystems.com, or by phone at 703-359-8160.


All OC Systems product documentation is available at 

The RootCause Console 'Help' menu points to the OCS Web site, as
well as the full documentation delivered with the product.


Product Installation and Licensing is described in Chapter 2 of the
RootCause User Guide and in the file "install.txt" located in the root
directory of the CD. 

Once installation is complete, you must source the RootCause/Aprobe setup
script to use the product. To do this, run:

  . <install_dir>/setup             (for ksh)
  source <install_dir>/setup.csh    (for csh)

After you have successfully installed RootCause, you should work
through the Demonstration Program described in Chapter 5 of the
RootCause User Guide.

If you are already a user of Aprobe, you can assume that this version
works the same.


This release is source-code compatible with previous
versions, so you shouldn't need to change any APC files. However,
this version is NOT binary-compatible, with respect to UAL files.  
That means you must recompile your probes from their APC source.

This version of RootCause/Aprobe runs only on AIX 5.3 or newer.  If
you need it to run on an older version, please contact OC Systems.

This version of RootCause/Aprobe is only applicable to 32-bit AIX
(xcoff32) applications.  Aprobe64 for AIX is available separately.

The RootCause Console requires Java 1.5 or higher. 

RootCause Intercept

The RootCause "intercept" capability (rootcause on) is not fully
implemented for AIX.  It is approximated by copying or soft-linking
a special script in place of any program to be intercepted.  
See the User's Guide for details.

Dynamically-loaded libraries

Tracing a "dynamic module" C/C++/Ada library causes it to be
statically loaded at program start (via aprobe using -dll command).
This means that one must be sure that the library specified in the
RootCause workspace must be the exact same one which the program would
normally load.  Furthermore, this won't work at all for libraries
whose initialization depends on program state. Any -dll libraries that
cause libc.so to be loaded at a different address than defined in the
program header will cause problems.
Java probes written completely differently than native (C/C++/Ada)
probes.  See the User's Guide and/or contact support@ocsystems.com about
probing Java.


You may be interested in these features/changes introduced in 
prior releases of RootCause and Aprobe.

Features and Fixes in RootCause 2.3.4 (AIX32 platform)
 (2691) add probe to log memory for hex dump at format
 (2692) create a call path profile probe
 (2695) memcheck does not record C++ allocations
 (2698) Apcentry crashes handling PA variant records
 (2699) add AP_MAKE_CPP_STRING to cppstring.h/c
 (2706) traced inlined code line numbers confusing
 (2713) remove help output from coverage output
 (2716) AIX can't set member field in class type
 (2720) xlC base class offset bug (too large)

Features and Fixes in RootCause 2.3.3 (AIX32 platform)
 (2667) AIX symbol demangling fooled by file sec.C
 (2668) Allow patching PPC trap instructions
 (2669) AIX Aprobe can't find start address in stripped exe
 (2677) AIX variant record size computed incorrectly
 (2678) Create probe to track (large) memory operations
 (2679) hmtb.ual doesn't show tracebacks
 (2682) Set maxdata for apmkadi

Features and Fixes in RootCause 2.3.2 (AIX32 platform)
 (2637) Coverage snapshot missing for forked pids
 (2639) AIX: can't probe stripped executable (Adaentry)
 (2640) statprof report improvements
 (2641) aprobe core dump on Xbarco Gcc app
 (2645) Create apcgen -U type dictionary.
 (2647) hmp report label MaxOutstanding not clear
 (2649) create HMTB - heap memory traceback probe
 (2651) prototype using a overloaded function selector

Features and Fixes in RootCause 2.3.1 (AIX32 platform)
 (2219) More helpful error message(s) when no reachable patch area
 (2582) Aprobe AIX does not demangle PowerAda operators
 (2592) COVERFILE should exclude (more) compiler generated func
 (2604) enhancements to memcheck: -S -e -E options
 (2610) Add pragma trap_patch to allow a trap patch on PPC
 (2613) provide mechanism to register dll/so with Aprobe
 (2616) small return type does not set entire reg
 (2617) branch coverage report should show which tests take branch
 (2619) Linux/AIX ap_StubRoutine on_line/offset does not work
 (2620) memcheck should support resetting freed block values
 (2621) Add indication of inlined function to apcgen -D
 (2628) track taken test names in brcov

Features and Fixes in RootCause 2.3.0 (AIX32 platform)
 (2552) statprof -a timings off due to bogus extra entry
 (2554) AIX boolean out parameter not set right on_exit
 (2558) COVERFILE should exclude compiler generated functions
 (2559) include date in statprof timestamp
 (2560) APO file doesn't handle single quote for options
 (2561) coverage does not do branches for some functions
 (2564) struct with ap_TimeT fields get false unions
 (2566) out of memory reading XCOFF to patch program exit actions
 (2572) add heap memory profile (hmp) probe

Features and Fixes in RootCause 2.2.9 (AIX32 platform)
 (2511) add ability to prune ADI files 
 (2517) merge brcov into coverage and extras
 (2521) add early Aprobe debug
 (2527) default name matching for .adi files not convenient
 (2528) apbc should process similar .brc files
 (2529) memleak and memwatch should have delayed start
 (2530) run axdln on .brc file when written by apformat
 (2531) apcgen reads all debug even if -f used
 (2532) AIX sigaction instrumentation causes memory problem
 (2533) coverage.ual core dumps if -if used
 (2535) apbc doesn't handle similar .brc files with .eab
 (2538) axdln -b foo.bc.orig doesn't work
 (2539) provide option to set statprof interval
 (2540) Upgrades to apcgp.
 (2541) abrmerge should warn if source file not found
 (2544) ppc branch coverage causes core dump with -O
 (2545) add statprof option to profile address range

Features and Fixes in RootCause 2.2.8 (AIX32 platform)
 (2492) axdln doesn't tolerate # lines
 (2493) apc rejects -unit name with mixed case
 (2494) apc should prefer local symbol if file scope given
 (2495) Further branch coverage improvements
 (2497) apcgen -T and -M don't work
 (2498) AIX _exit() from thread doesn't trigger program on_exit
 (2505) create a name demangling tool
 (2508) AIX apcgen prints wrong decl file/line

Features and Fixes in RootCause 2.2.7 (AIX32 platform)
 (2068) apc+PA+AIX: apc size warning for 2d array
 (2389) AIX apc reports MP_TREE_COLLISION
 (2449) brcov doesn't handle lines from multiple files
 (2451) support enable/disable calls and apdemand in memstat
 (2453) brcov improvements
 (2454) COVERFILE files cause core dump when inserting multiple files
 (2457) apbc.pl doesn't correctly mark some lines
 (2458) AIX Apcentry ap_StrFree coredump in CloseCoffModule
 (2459) AIX Apcentry coredump in SetCurrentStatement
 (2460) apcgen warnings for C++ symbol _Z
 (2461) apcgen -La reads all debug info
 (2462) apbc/bc.pl does not handle unprobed lines in .bc
 (2467) apcgen and apc should declare maxdata:0x40000000
 (2472) typo in error message
 (2474) Aprobe setup script must check for all of Aprobe
 (2475) deliver AdaDI shared library with AIX Aprobe
 (2484) -qmin_heap breaks memstat traceback table

Features and Fixes in RootCause 2.2.6d (AIX32 platform)
 (2053) probe calling ap_FileExists causes Ada exception
 (2374) $APROBE/setup should update library path
 (2375) Warn if probing system calls
 (2376) log fixed-length string return value fails
 (2378) partial coverage percentages not obvious
 (2388) atcmerge should print Aprobe version
 (2394) support wildcard in apcgen -p argument
 (2395) support logging local/global variables in apcgen
 (2396) make ExpandDottedLines the default for coverage
 (2397) support COVERFILE 'sourcefilename' in coverage.cfg
 (2400) axdln -s doesn't work on specs
 (2401) Coverage should report reasons functions not probed
 (2404} Warn about mismatched function version in coverage
 (2405) atcmerge should allow merging diff func versions
 (2406) apbc should warn if no source files found/given
 (2409) atcmerge should produce .bc file as an option
 (2411) Allow aliases of module names (libc.so.6)
 (2415) Coverage should report maximum offset count for line count
 (2423) Add source file matching option to apcgen
 (2424) apcgen generated incorrect scope for func-local var
 (2426) Problem stubbing abort() with coverage.ual
 (2427) Need mechanism to specify function not to sacrifice
 (2430) provide access to STL exception message
 (2431) exceptions.ual should provide Ada/C++ exception message string
 (2437) on_exit probe breaks cross-lang exception raise
 (2439) ppc sacrifice function with large TOC
 (2441) memstat statistics are displayed incorrectly
 (2443) provided traceback blacklist address support

Features and Fixes in RootCause 2.2.6c (AIX32 platform)
 (2368) AIX coverage.ual reports empty non-executed lines for 
 (2370) AIX: wrong symbol filename used due to #include

Features and Fixes in RootCause 2.2.6b (AIX32 platform)
 (2320) AIX C++ coverage included lines/files wrong

Features and Fixes in RootCause 2.2.6 (AIX32 platform)
 (2268) can't set float return value in on_exit after stub
 (2276) AIX apinfo core dump AddressOfCallToMain
 (2303) AIX atcmerge core dump, 0 length FT (E74103)
 (2304) axdln doesn't handle nested generics
 (2305) AIX apcgen -D prints wrong line number for symbol
 (2308) AIX fail to handle Ada function named operator (E74146)
 (2309) AIX axdln must tolerate C++ symbols and old-style input
 (2313) AIX C++ global symbol not demangled
 (2314) apinfo -h dumps core on AIX
 (2317) deliver call graph probe apcgp.ual

Features and Fixes in RootCause 2.2.5d (AIX32 platform)
 1. (1361) add changes, comments to coverage.2/bc.pl
 2. (1477) coverage sometimes fails to report missed lines
 3. (1516) AIX aprobe reports wrong lines for inlined C++
 4. (2203) coverage.cfg should allow inline comments
 5. (2221) AIX apc logs wrong registers for xlC/ C++ parameters
 6. (2228) implement trap patches on AIX PPC.
 7. (2240) no coverage snapshot on SIGINT or exception
 8. (2241) atcmerge fails merging .tc files from multiple apps
 9. (2242) apcgen -f doesn't handle include files
10. (2244) apcgen misses C++ members fields in big class
11. (2249) optionally reset counts between coverage snapshots
12. (2250) each coverage snapshot should write separate .tc and .bc
13. (2253) coverage summary report line should put symbol last
14. (2254) coverage CommaOutputFile gets duplicate content
15. (2255) implement ap_LoadAllSymbols for AIX...
16. (2256) improvements to coverage reports
17. (2261) add experimental expand-dotted-lines tool
18. (2269) enhance statprof to profile entire address space
19. (2270) report all default options in coverage cfg file

Increasing patch capacity

Several improvements to Aprobe for AIX were made in Aprobe 4.4.5d to
increase the number of patches that can be applied, mainly to support
coverage testing on lines.  Two options will increase the (limited)
base capacity for patching:

-qenable_trap_patches :

This option enables trap patching, which, while slightly slower than normal
branch patching, avoids the restriction that patch space must be reachable
from the patched function.  Patch areas may be located anywhere in the address
space rather than within reach of or within the application module.


This option enables Aprobe to allocate new patch space if the base patch
area is exhausted.

Features and Fixes in RootCause 2.4.5c (AIX32 platform)
 1. (2174) aprobe cores loading UAL that depends on a .so
 2. (2176) aprobe should give more info when ual fails to load
 3. (2179) update AdaUnboundedString.h for PowerAda i2172
 4. (2195) ADI files from same src should be bitwise identical
 5. (2205) problem parsing gcc 5.x attribute list
 6. (2207) ap_DoCoverageSnapshot() doesn't work
 7. (2217) apc driver should not invoke user's .kshrc file
 8. (2218) Provide a way to increase density of patches on AIX
 9. (2219) More helpful error message(s) when no reachable patch 
10. (2227) system_ual.apc cannot compute StartAddress

Features and Fixes in RootCause 2.4.5b (AIX32 platform)

 1. (2139) lines not logged if duplicate timestamp
 2. (2141) apc parse error parsing complex gnu C type decl
 3. (2145) AIX crash starting IBM MB with aprobe
 4. (2146) AIX aprobe on java6_64 causes memory corruption
 5. (2152) AIX apcgen doesn't recognize .o as being C++ if -qnotimestamps
 6. (2155) optionally disable __ThrowV6 probe
 7. (2160) fix memwatch, memstat examples, output
 8. (2161) java_memstat, java_exceptions no longer supported
 9. (2164) java aborts under RC trace

Features and Fixes in RootCause 2.4.5a (AIX32 platform)
 1. 2117 show instantiation line in apc/apcgen comments

Features and Fixes in RootCause 2.4.5 (AIX32 platform)

 1.(1182) on_line(last) doesn't work for generic instance
 2.(2069) unhelpful message for $a[i, j]
 3.(2074) AIX RC Trace Setup doesn't show lines, modules
 4.(2108) adapt aprobe to PowerAda i2098 change
 5.(2114) AIX apjava doesn't recognize java7

Note about i2108: AIX PowerAda 5.8 introduced a small change in stack
layout in order to fix an incompatibility with the AIX ABI.  This change
breaks upscope data references in probes built with Aprobe 4.4.4a and
earlier. That is, if you're logging Ada variables in your probe, and
you want the probe to work with PowerAda 5.8, then you have to install
and use version 4.4.5a or newer.

Features and Fixes in RootCause 2.4.4a (AIX32 platform)

The focus of this delivery was compatibility with AIX 7.  
This version also introduces a new "demand.ual" predefined
probe, and fixes some RootCause UI problems.  Specifically:

 1. (1966) apjava fails on latest IBM Java5
 2. (2023) aprobe doesn't install gcc on aix6,7
 3. (2028) apc can't parse unistd.h on aix7
 4. (2030) rootcause crashes on AIX7 expanding modules
 5. (2034) add a general user demand interface to apc
 6. (2036) memstat gui closes prematurely
 7. (2037) AIX RC 'Add Dynamic Library' should handle libX.a(shr.o)
 8. (2038) Update RC/Aprobe FAQ 
 9. (2044) AIX apc gens wrong code to deref short C++ ref param


A new predefined probe, "demand.ual", along with its header file
demand.h and supporting command-line tool apdemand, provide a
framework for "demanding" action from another probe, from the
command-line at run-time, independent of what the probed program might
be doing.  This is an advanced feature, but can be powerful in the
right circumstances.   To learn more, copy the example directory
$APROBE/examples/predefined_probes/demand to a working area, and start
with the README file there.  It shows how you can use demand.ual in
conjunction with profile.ual to take a performance snapshot 
"on demand".  The "RemoteControl" file in that same directory
provides more detail in how to use this to control your own probes
and applications.  Contact support@ocsystems.com if you have questions.

Features and Fixes in RootCause 2.2.3b (AIX32 platform)
 1. (2002) PA eqOp symbol demangled wrong by RC223a

Features and Fixes in RootCause 2.2.3a (AIX32 platform)
 1. (1960) apmkadi takes a long time on new AIX/C++
 2. (1962) (OCS) don't retry failed demangling
 3. (1964) (OCS) provide libprobe.adi

Features and Fixes in RootCause 2.2.3 (AIX32 platform)

 1. (1896) apc -v shouldn't do gcc -v
 2. (1912) redefinition of typedef .. PtrTo_void
 3. (1930) provide C++ string helper UAL

 All of the above fixes also apply to the 64-bit version of the product.

 Issue 1930 introduces "cppstring.ual" which provides ap_GetCppString()
 and ap_SetCppString() functions for use in your probe.  Its usage is
 illustrated in $APROBE/examples/predefined_probes/cppstring/.

Features and Fixes in RootCause 2.2.2h (AIX32 platform)

 1. (1391) bad layout for C++ struct
 2. (1432) bad APC layout for Ada Recon_Table_List_T array
 3. (1673) memstat totals must be 64 bits
 4. (1688) 'offsetof' w/gcc4 breaks apc parser
 5. (1702) regression in apc type layout for Convert_Route_Sr
 6. (1707) 'typeof app_type apc_type' ignored in ERAM app
 7. (1718) ApcDI shouldn't call back for powerada debug info
 8. (1724) ap_DoPeriodically() should be cancellable
 9. (1738) parse error for c99 restrict keyword
10. (1739) adapt to xlc10 
11. (1771) unhandled sm_class (22)
12. (1772) huge array generated for xlC 11.1 variable
13. (1783) regression in tasltrc-aix.apt
14. (1786) C++ double fields not correctly aligned

Features and Fixes in RootCause 2.2.2f1 (Aprobe 4.4.2f1)
 1. (1639) core in utb_RegisterAprobeFrameCallback()
 2. (1643) provide probes to use AIX hardware watchpoints
 3. (1660) bad branch in call patch crashes g1.eab
 4. (1689) apc crashes due to gcc4 internal error (patch f1)

Issue 1643 introduces a new predefined probe, traphwp.ual.  
Documentation for this is in $APROBE/probes/traphwp.README. 
An example is in directory $APROBE/examples/predefined_probes/traphwp.

Features and Fixes in RootCause 2.2.2e (Aprobe 4.4.2e)
Version *.2.2e was not released for AIX.  
The following changes from that version apply to the AIX platform:

 1. (1335) apcgen -D should force -d
 2. (1551) deliver memleak.ual
 3. (1574) AIX: add jvmti-based java probes
 4. (1581) AIX: perf_cpu.ual not getting delivered
 5. (1582) AIX: aprobe.jar has classes not in com.ocsystems
 6. (1586) AIX: update Aprobe's gcc to 4.x, use -gxcoff
 7. (1588) AIX: regression in PowerAda variant record accesses
 8. (1592) AIX: apsymbols should not apply -x on AIX
 9. (1617) Make traceback_compression compilable by user
10. (1621) ap_DoPeriodically incorrectly uses signal on threaded app
11. (1636) fix problems with log_env probe

Issue 1551 introduces a new predefined probe, memleak.ual.
Documentation for this is the source for the probe itself,
$APROBE/probes/memleak.apc.  A simple example is in directory

Features and Fixes in RootCause 2.2.2d (Aprobe 4.4.2d)
1.  (1382) apcgen crash listing overloaded symbols
2.  (1508) Update examples/predefined_probes/coverage.2/bc.pl
3.  (1510) (AIX) AIX 5.3 TL6 enforces PT_MULTI  
4.  (1513) build and deliver $APROBE/lib/libdal.so.stub
5.  (1514) apformat symbol lookup crashes in split PA csect
6.  (1528) deliver $APROBE/lib/libdal.exp (see i1513) 
7.  (1539) (OCS) adapt to PA i1385 export preamble size change
8.  (1540) (OCS) wrong or no line number for exported PA routines

Features and Fixes in RootCause 2.2.2c (Aprobe 4.4.2c)

The following changes and fixes are included in this release that
were absent from the previous one.

 1. (1189) apc fails on module with bad/missing supporting module
 2. (1207) apformat should remember ADI files (see NOTE 1)
 3. (1360) Allow user to specify patch saving mode (see NOTE 2)
 4. (1386) (OCS) fix string leak in CopyNode
 5. (1401) xreraise not found while doing coverage
 6. (1404) add apexc v3 support (see NOTE 3)
 7. (1415) Java 1.4.2 has libjvm.a in sovvm directory
 8. (1429) Aprobe should save/restore FPSCR
 9. (1431) Remove sovvm as built-in JVM (see NOTE 4)
10. (1434) (OCS) provide workaround for bad type layout (see NOTE 5)
11. (1436) (OCS) refactor type replacement implementation
12. (1441) (OCS) implement 'formatof' to manipulate apc print routines
13. (1442) different apc results for upper, lower case
14. (1451) aprobe must register with utb at startup
15. (1465) (OCS) apexc regression misses subprograms
16. (1467) $result_mask not found when many probes in 1 file
17. (1484) fix regression with Ada field names
18. (1485) formatof + typeof don't work together
19. (1487) -x module path must override LIBPATH (see NOTE 6)

The following notes explain new features introduced in 2.2.2c/4.4.2c:

  NOTE 1: apformat now "remembers" ADI files (i1207)
  By customer request, aprobe logs the paths of ADI files that were
  loaded at runtime, and apformat attempts to load these again at
  format-time.  apformat will not fail if they are absent, and apformat
  can be forced not to load them by renaming the .adi file itself, or
  specifying -z on the apformat command line and explicitly naming UALs
  that are wanted.  Also, apformat -u info -p "-u" ... now lists ADI
  files applied at runtime.
  NOTE 2: New aprobe -q conserve_patch_space option added (i1360)
  If you are getting errors like "Could not allocate a patch area
  reachable from address 0xd1234560", you can try re-running with -q
  converse_patch_space.  This attempts to use smaller (16-byte) patches
  which branch to allocated memory rather than putting the entire
  (80-byte) patch in the code section, which may enable all your patches
  to fit, at the expense of a slight slowdown in execution.
  NOTE 3: Apexc version 3.0 introduced (i1404, i1465)
  Aprobe's exception analysis and call graph generation tool was
  significantly enhanced earlier in the year at customer request,
  and is included in this release.  Usage has changed significantly:
  see $APROBE/apexc/ApexcUserGuide.doc or ApexcUserGuide.htm.
  NOTE 4: Probing Java on AIX 5.3 fixed (i1415, i1431)
  The IBM /usr/java14/bin/java command analyzes the machine its running
  on, and selects which of several JVM libraries to load in an undocumented
  manner.  RootCause/Aprobe must know this library ahead of time in order
  to apply probes.  If Java probes don't seem to be working for you,
  you can do the following in your environment and re-start your java
     export APROBE_OVERRIDE_LIBJVM=sovvm
  If this doesn't resolve your problem, contact support@ocsystems.com.

  NOTE 5: APC auto-formatting extensions added (i1434, i1436, i1441)
   This version introduces two extensions to the APC language to allow
   user-specified replacement of default apc behavior.
   ** typeof Declaration **
   The first extension is a new use of the keyword 'typeof' to designate
   a locally defined APC type to be used in place of the type that the
   apc compiler would generate to overlay the in-memory image of a
   target program's type.  For example:
     typedef struct
        int word0, word1;
     } my_time_type;
     typeof $time_type  my_time_type;
   Then wherever an object or field of type $time_type was logged, it
   would be interpreted as type my_time_type.
   Why would you do this?  Mostly to work around bugs or limitations in
   the APC compiler.  The above is a real-world case where time_type was
   defined as a 'double' but the 8-byte alignment imposed by the apc
   backend compiler didn't match the 4-byte alignment used by the target
   compiler.  By replacing the 'double' with a struct consisting of 2
   ints, the alignment restriction was removed and everything worked.
   The onus is entirely on the user to define a compatible type; there
   is no checking for compatibility.  The original type layout must be
   determined by looking at the C code generated by the 'apc' compiler:
     apc -g t.apc -x t.exe  # creates t.apc_c.c file
   This can be further cleaned up and made readable, for example:
     cat t.apc_c.c | \
       grep -v "^#line" | \
       sed "s/^ *//" | grep -v "^$" | \
       indent -st -sob -dj -i3 -bl | expand -3
   It can still be complicated, so generally the replacement type will
   be determined in cooperation with OC Systems support.  This can then
   be defined in an include file that users can include into their
   probes as needed.
   ** formatof declaration **
   The typeof extension described above led to the need for a
   corresponding formatting extension, so that the replacement type
   could be printed as if it were the original, or in any other way the
   user desires:
      formatof type_name print_proc_name;
   This declaration says: when generating the automatic formatting code,
   call the already-defined function 'print_proc_name' to print any 
   object or field of type 'type_name'.   A "print proc" always has the
      void print_proc_name(int Indent, ap_NameT Prompt, type_name *Arg);
   For example, see the functions ap_Print_* in aprobe.h.  So, to use
   the formatof declaration to print "my_time_type" as a double value,
   you would do something like:
      static void 
      Print_my_time_type(int Indent, ap_NameT Prompt, my_time_type *Arg)
         double d;
         memcpy(&d, Arg, sizeof d);
         ap_Print_double(Indent, Prompt, d);
      formatof my_time_type Print_my_time_type;
   As with typeof, there is another form of 'formatof' which gives
   the existing print procedure name for a given type or expression:
      formatof( type_name )
   So you could replace the call to ap_Print_double above with:
      formatof(double) (Indent, Prompt, d);
   Yes, it looks strange, but it works.
   As with 'typeof', it's up to the user to ensure that the print
   routine is declared with the right prototype and does something
   sensible.  As with format routines the Arg parameter is passed by
   reference as in the example above.
   The major limitation we know of right now is that the 'formatof'
   mechanism doesn't work for array types, even fixed-sized ones.  
   This is because of the special handling of such types to capture the 
   size of an array as a "slice".  
   Aprobe historians may know that 'formatof' was originally defined
   as part of the language, for actually defining format routines.
   While the syntax was recognized, it was never implemented because it
   didn't fit the general usage of automatic formatting.  This seemed
   like a good way to capture the original intent.

   NOTE 6.  Issue 1189 fixed some problems with reading supporting
   modules from explicitly provided module, including the problem where
   LIBPATH was ignored.  Issue 1487 fixed a related issue where a
   supporting module PATH overrode a path from the command-line in
   certain cases.  The result of these fixes is to implement the
   original intended behavior of apc: that if m.exe references libt1.so
   (as shown by ldd or dump -H), and your probe t1.apc contains 
   'probe "foo()" in "libt1.so"', then 'apc t1.apc -x m.exe' will 
   find libt1.so and load it just as when you run m.exe.

Features and Fixes in RootCause 2.2.2a (Aprobe 4.4.2a)

NOTE: RootCause 2.2.2b (Aprobe 4.4.2b) was not released, so its
      changes are documented above as part of the current release.

The following changes and fixes are included in this release that
were absent from the previous one.

 1. (1216) PowerAda exception & shared runtime issues (See NOTE 1 below.)
 2. (1324) apexc doesn't correctly match class-wide catcher (See Note 2 below.)
 3. (1328) AIX aprobe gives misleading 'saprobe' message
 4. (1333) apmkadi v4.4.2, xlC8.0.0.11 crashes on cc.eab
 5. (1336) PA line table read wrong for shr.o in archive
 6. (1344) update apexc docs and official name
 7. (1347) (F) Aprobe internal error...xreraise() not found
 8. (1354) install_rootcause doesn't set gcc path

  NOTE 1: PowerAda Exception Support
  Support for ap_RaisePowerAdaException in the context of PowerAda shared
  libraries was added.  Most of this is not user-visible, except that
  operations now work that didn't before.  However, a new macro was provided
  to allow the user to specify the shared library in which an exception
  is defined, if it is not present in the module being probed:
     ap_RaisePowerAdaExceptionInModule(Name, Module)
  for example:
      ap_RaisePowerAdaExceptionInModule("App_Error", ap_ApplicationModuleId());
  NOTE 2: apexc now fully supports C++
  While Aprobe 4.4.2 included most of the C++ support for apexc, there
  were still some problems, which are now resolved, and apexc may be
  analyze C++ exception usage.

Features and Fixes in RootCause 2.2.2 (Aprobe 4.4.2)

This delivery includes the following fixes/features not in previous
releases or patches:  

 1.  (970) report error calling ap_Free() with an ap_StrDup'd pointer
 2. (1014) Execute user-provided prebuild.ksh, postbuild.ksh in RC workspace
 3. (1108) add ap_StackTop() to aprobe.h
 4. (1123) timing window in ThreadContextPtr management.
 5. (1135) apc crash: nodetype.c, line 270: Unexpected Operation ... 
 6. (1144) probe thread nested functions may get wrong thread context
 7. (1146) Aprobe: Aix: target mem ops for large blocks
 8. (1150) #pragma xyz { fails with parse error
 9. (1154) ap_DoPeriodically is unreliable in signal mode
10. (1155) Java Class Verification Error - duplicate names
11. (1157) \r is not handled by the probe config file reader
12. (1174) aprobe gets wrong size for powerada array
13. (1181) add apformat -f support (see NOTE 1 below)
14. (1196) apcgen -x shared_library should use correct shared lib name
15. (1209) verbose output from aprobe tools should go stderr (see NOTE 2 below)
16. (1215) ADI-mismatch warning not given for shared libraries
17. (1217) sigsegv should treat SIGABRT as 'fatal'
18. (1222) apc crashes against optimized code
19. (1231) periodic thread may get context of another thread
20. (1238) add GNAT support to AIX Aprobe (see NOTE 3 below)
21. (1240) implement AIX gcc/GNAT uplevel references
22. (1250) rootcause_install prompt should mention Ada
23. (1258) bad param passed to nsleep() by ap_Pause
24. (1259) adapt aprobe assembly to PowerAda 5.5 
25. (1281) aprobe crash on pa5.5 exported proc
26. (1282) apcgen emits bad APC for GNAT record param
27. (1289) apc logs bad string size for opt. PA param (see NOTE 4 below)
28. (1290) aprobe problems on PA5.5 pop_stack_and_call*
29. (1291) apexc cores on c5.eab
30. (1299) no line info for export PA5.5 C routines
31. (1303) avoid race condition in ap_HashTableDelete
32. (1310) re-enable 'DisplayReport SubtotalReport' in events.ual

  NOTE 1: Using 'apformat -f'
  The -f option to apformat, introduced in Aprobe 4.4.2, indicates that
  apformat should run in follow mode, similar to "tail -f", in which APD
  files are continuously read and formatted as they are written.
  Formatting starts at the file specified by the -n option, just as it
  does without -f:
     - By default (-n not specified), formatting starts at the beginning 
       of the lowest (oldest) undeleted ring file.
     - If n > 0, formatting starts at the specified number, e.g., 
       apformat -f -n 2 t.apd starts formatting at t-2.apd if available.
     - If n == 0, formatting starts at the beginning of the file
       currently being written (the newest).
     - If n < 0, starts at newest + n, that is, if t-4.apd is newest,
       then 'apformat -f -n -2 t.apd' will start formatting at t-2.apd.
  In all cases, apformat will wait for new data to arrive, even if the
  probed process is no longer running.  You can cause apformat to exit
  at the end of data by logging on program exit using a format that calls
  "_exit()", as follows:
     static ap_BooleanT ExitFormatsCalled = FALSE;
     probe format
           ExitFormatsCalled = TRUE;
     extern void ap_RunFormatExitProbes();
     static void ProgramExitFormat()
        if (!ExitFormatsCalled)
           // Force the exit to avoid
           // Flush the buffers and exit.
           // Use _exit(), instead of exit(), so that atexit() 
           // actions would not get triggered again.
     probe program
           // Record the fact that we are exiting the program.
           // This will allow us to stop formatting in apformat -f (tail) mode.
           log() with ProgramExitFormat;

  NOTE 2: Debug, Verbose Output Changes
  Output written as a result of the following operations is now written
  to standard error, whereas before this was written to standard output:
    - specifying options -! or -v to aprobe and apformat
    - calling ap_Debug()
    - calling ap_DumpMemory()
  NOTE 3: Support for GNAT Applications on AIX
  This version introduces support for GNAT-compiled programs on AIX.
  All support is as described for GNAT Ada programs in the user guide,
  with the following exceptions:
      - exceptions raised by signals don't show a good traceback.  Use the
        sigsegv.ual predefined probe to get reliable signal traceback.
        This is a permanent limitation.
      - ap_RaiseGnatException is only supported on_line, not on_entry or 
        on_exit, and the traceback information reported for an exception
        raised by ap_RaiseGnatException is not reliable. (i1313)
  NOTE 4: Logging Parameters on_exit
  In earlier versions, an attempt was made to log scalar (in-mode,
  pass-by-value) parameters on_exit from their location on the function's
  stack, in order to pick up any changes the user code may have made.
  However, in the presence of optimization, parameter values are often not
  stored and bad values were reported.  Under issue 1289, 'on_exit log($1)'
  will always log the value of the first parameter as it was saved on_entry
  _unless_ the user compiles the apc with the new option, "-y", which forces
  it to log what's actually in the parameter's canonical location on_exit.
  You'd really only use -y if you wanted to get the correct value of "i"
  on_exit to something like the following: void foo(int i) { i++; }

Features and Fixes in Aprobe patch 4.4.1a1 

 1. 1251 handle PA _AnonField as WrapperFields
 2. 1254 apc crashes in ParseTypeXCOFF
 3. 1261 xlc8 puts '<' in stabstrings
 4. 1263 handle 'empty' base class from xlC8

Features and Fixes in RootCause 2.2.1a/Aprobe 4.4.1a

The purpose of this patch delivery is to support programs compiles with
IBM xlC8.0 on AIX 5.3.  This delivery includes fixes for the following

 1. (1109) xlCBind+AdaSym'Access+ADI = bad ap_FunctionPointer(AdaSym)
 2. (1171) apexc crashes on df.eab
 3. (1220) handle programs compiled with xlC 8.0
 4. (1226) fix crash in libdal after signal 
 5. (1227) libdal waitpid() should retry on EINTR
 6. (1232) AIX: xlC8 doesn't list compiler param in stabs
 7. (1236) handle "# pragma" found in xlC8 header files

Features and Fixes in RootCause 2.2.1/Aprobe 4.4.1

 1. (1000) sacrifice/relocation of yylex crashes cc.eab
 2. (1009) on fork, patch server process calls atexit()
 3. (1010) FormatAddressSize may be insufficient
 4. (1011) probe-based call/exception reporting tool (Note 1)
 5. (1013) ap_FormatAddress() crashes for lines
 6. (1018) make APROBE_DUMP_CORE_IF_ERROR more useful
 7. (1029) incorrect layout of dynamically-sized PowerAda type (Note 2)
 8. (1043) stack overflow due to bad sacrificed function size
 9. (1047) apc NO_BIND_INFO crash reading C++ main Ada modules
10. (1056) enhance info.apc to dump address, size for symbols
11. (1081) GCC cplus_demangle leaks memory
12. (1094) sacrifice of libjvm 1.4 symbol cores apjava
13. (1097) extra white space in formatted output
14. (1101) aprobe -if -u trace prints null timestamps

  NOTE 1: apexc tool introduced

  AIX Aprobe 4.4.1 includes "apexc", an Aprobe-based tool for
  building a database of call and exception propagation information for
  PowerAda programs.  For full information see
  $APROBE/apexc/ApexcUserGuide.htm or ApexcUserGuide.doc.
  NOTE 2: PowerAda version compatibility 

  Part of the fix for i1029 is in PowerAda 5.4d, and requires
  that version or higher.  A number of other bugs reported against
  Aprobe's behavior with PowerAda programs were fixed in PowerAda 5.4d
  as well:
    514 warn if multiple powerada libs in single apc command
    701 order of ada probes causes 'no debug available'
    906 apc chokes on inst of bc hash_table
   1043 stack overflow due to bad sacrificed function size
   1047 apc NO_BIND_INFO crash reading C++ main Ada modules

Features and Fixes in RootCause 2.2.0/Aprobe 4.4.0

This is a major version change and as such introduces Aprobe API
changes.  The primary focus of this delivery is reduction of Aprobe
memory usage.  There are no new features in the RootCause graphical user

 1. When running under RootCause (with Aprobe), Aprobe now adds 
    about 1/4 as much memory overhead as in the previous version.

 2. At runtime, RootCause (Aprobe) now gets its memory from the application
    heap by default, rather than using private memory-mapped regions which
    are wasteful of virtual memory.  An environment variable 
    APROBE_MMAP_HEAP may be defined to force use of mmap for Aprobe memory,
    but this is not recommended.

 3. A new flag, "-qmin_heap" may be specified which reduces memory 
    usage much more for large applications with many symbols, but which
    disables some operations at run-time.   See "Memory-Reduction Options"
    below for more details.

 4. A new tool, "dumpxcoff" is provided as an aid in analyzing the 
    content of AIX xcoff modules.

 5. The mechanism for disabling/enabling probes has been overhauled
    with more reliable mechanisms.   See "Disabling Probes" below.

 6. The file $APROBE/lib/aprobepatcharea.o is delivered for inclusion in
    your applications.   Just add
        $APROBE/lib/aprobepatcharea.o \
    to your link (xlC or ld) command to include a patch area to avoid 
    "No patch area reachable" warnings.

Bug Fixes in RootCause 2.2.0/Aprobe 4.4.0

 1. Message "Double field size and/or alignment cannot be reproduced by apc"
    is gone; the PowerAda long_float field is correctly logged. (i700)

 2. Two new macros, "BeforePoweradaCall" and "AfterPoweradaCall" are provided
    to allow PowerAda calls to be made from within Probes. (i722)

 3. Crash during PowerAda exception propagation across probes due when using
    Ada shared libraries is fixed. (i823)

 4. References from probes to PowerAda package static data no longer
    contain module-specific numeric offsets. (i855)

 5. A failure to instrument the correct malloc() variant in the memwatch and 
    memstat probes on AIX 5.3 was fixed. (i871)

 6. A problem with automatically referencing a user-provided 'ap_patcharea'
    definition was fixed. (i872)

 7. apcgen no longer generates the syntactically-incorrect '$virtual-base-ptr' 
    for C++ methods (i887).

 8. The operation "Find Function in Trace Events" now works for Summary nodes
    in the RootCause Trace Display window. (i902)

 9. The "bc.pl" script provided in examples/predefined_probes/coverage.2 
    no longer requires a full path name to the Ada source file, and the
    documentation has been updated to reflect Ada limitations. (i904)

10. A boundary-condition problem with long names in ADI files was fixed. (i754)

11. Prevent duplicate symbol recording in ADI files. (i756)

12. Correctly handle (ignore) deleted xcoff debug symbol entries. (i815)

13. Provide $APROBE/include/AdaUnboundedString.h for logging these. (i827)

14. Unmap ADI file after reading (i844) and writing (i829).

15. Correctly handle xcoff bigtoc patch ("@FIX") symbols. (i836)

16. Ignore duplicate symbols created by -qfuncsect switch. (i839)

17. Avoid gcc bug in system_ual.so causing PTHDB_CALLBACK error (i840)

18. Avoid crash when using aprobe -! option. (i842)

19. Avoid duplicate fieldnames caused by multiple virtual inheritance. (i881)

Features and Fixes Introduced in RootCause 2.1.4b
 1. Corrected a problem where incorrect code was generated by apc for
    a PowerAda record type (i36)

 2. Corrected a problem where incorrect code was generated by apc for
    accessing members of a class with multiple parent classes some of
    which are virtual. (i89, i642)

 3. The libdal.so library is now included in
    rootcause_agent.tar.Z. (i523)

 4. A problem with correct adjustment of file references in a deployed
    workspace was fixed. (i624)
 5. A workaround was implemented for IBM PMR 46650.487, in which mmap
    failed to correctly read data from memory-mapped files.  The
    workaround is enabled by defining the environment variable
    APROBE_AIX53_MMAP_WORKAROUND to a nonempty value. (i631)

 6. The file $APROBE/probes/events.cfg was updated to reflect current
    implementation. (i638)

 7. A problem, in which probing parameters $8 and above for a PowerAda
    subprogram in which an earlier parameter is a
    dynamically-constrained packed bit array, was fixed. (i645).

 8. The function ap_GetHostId() was changed to return the same value
    as the "hostid" command instead of the license manager's host ID,
    which is now provided by the new function ap_HostLicenseId().  The
    function ap_GetHostName() which returns the same string as
    "hostname". (i647)

 9. The memwatch, memstat java_memstat probes now support "REMOVE"
    filters which identify allocations that should not be reported.
    This are documented in the sample configuration files for these
    probes provided in $APROBE/probes. (i650)

10. Several small runtime memory leaks were closed. (i668, i681, i720)

11. The memstat and java_memstat probes now copy the default
    configuration file from $APROBE/probes when none exists, rather
    than generating one internally.  There is no functional change,
    but it allows installation-wide customization of the
    configuration. (i674)

12. The ADI (Aprobe Debug Information) support has been improved in
    several ways: aprobe will look for the default-named adi file in
    the same directory as the application and will use it if it's
    checksum matches; apmkadi -v reports more information; aprobe -v
    will report whether ADI file was used and if not, why not (e.g.,
    checksum mismatch).  (i680)

13. An infinite recursion reading the debug information for a PowerAda
    recursive record type was fixed, but the fix requires PowerAda
    version 5.4b. (i694)

14. The time required to start RootCause on a large application was
    reduced by only reading partial information at startup. (i697)

15. The option -D_THREAD_SAFE is now implicitly provided when
    compiling apc files, to ensure the thread-safe value of "errno" is
    used. (i709)

16. The 'cc' and 'xlc' compiler profiles have been restored to the
    delivered compiler_profiles file, to allow probes on C and C++
    applications to be compiled with the application compiler for
    easier debugging. (i709)

17. A problem using thread-level probe data from a nested probe was
    fixed (i714).

18. A potential deadlock in use of ap_HashTable was removed. (i717)

19. The libdal library has been enhanced to tolerate multiple
    invocations.  This will allow an application to explicitly specify
    a negative priority to force libdal initialization prior to early
    application initialization with
    "-binitfini:ap_DalLibraryLoader::-100000000" (i718)

20. Better error diagnostics are provided if libdal.so fails during
    patching. (i724)

21. A problem in which some C++ exception propagation routines in
    libC.a were not being probed was fixed.  (i727).

Features in RootCause 2.1.4a

 1. The management of APD rings in the presence of frequent process creation
    was made more robust (i409).

 2. apinfo -l now prints the source file name associated with source
    lines (i538).

 3. Interpretation of reference parameters with function calls has been 
    reverted to the behavior prior to 4.3.4 (i578)

 4. PowerAda shared libraries are now supported by the RootCause 
    Trace Setup dialog (i581).

 5. A regression in the statprof probe was fixed (i584).

 6. A bug in logging PowerAda records with trailing pad bytes was
    fixed (i599).

 7. A crash caused by an unsupported xlC debug symbol was fixed (i604).

 8. A bug causing xlC line information to be lost was fixed (i605)

Features in RootCause 2.1.4

 1. The events probe was enhanced to export JNI calls for all functions defined
    in events.h.

 2. Improvements were made in the way dot-indicators are propagated
    in the GUI's Trace Setup window.

 3. AIX 5.3 is now supported by Aprobe.

 4. apc, apcgen now correctly find PowerAda debug information from
    shared library archive members like libutilities.a(ada_shr.o), and
    no longer invoke the 'nm' tool.

 5. A problem where Aprobe disturbs the global "errno" value was been

 6. A new option (-r) has been implemented for the statprof custom
    probe that resets the counts to zero for each snapshot. This will
    result in profiling results for each interval between snapshots
    which are independent of each other, rather that the default which
    is cumulative.

 7. Several corrections were made to the custom memcheck probe.

 8. A custom probe is provided to allow instrumentation of the
    de-facto standard for Java logging, log4j. This probe provides the
    ability to watch the configuration file to look for changes.

 9. Handling of very large Java methods has been improved.

10. A new function is added to aprobe.h:

       extern void ap_HeapStats(ap_Uint64   *TotalBytesAllocated,
                                ap_Uint64   *TotalBytesFreed);

    This may be called any time in a probe to determine the amount of
    memory allocated by Aprobe itself, separate from the application
    being probed.  When running aprobe with the -v flag, these
    statistics are printed at various points during program startup, for

       Aprobe: Initialized thread stack
       Aprobe: Heap memory stats
               Allocated bytes: 26016
               Freed bytes: 0

11. The Aprobe run-time system and probes, as well as any user probes
    that use ap_Malloc(), now allocate heap memory from shared memory
    allocated using mmap() rather than from the program's memory space
    using malloc().  This allows Aprobe to handle applications with huge
    symbol tables and memory needs.

    The aprobe configuration parameter "-qmmap=<address>" has been
    extended to apply to all uses of mmap including this heap memory
    allocation.  Hence the specification, "-qmmap=0xb0000000" on AIX
    would indicate that the first mmap operation would begin in segment
    11, and each subsequent mmap operation would request the next
    available page after that, whether used for APD files, aprobe heap,
    or another purpose.

    In order to allow all aprobe shared memory usage to fit into a
    single segment by default, the default size of a single APD file was
    reduced from 256M to 64M.  As always, this can be changed with the
    -s option and multiple APD files can be specified with the -s
    option.  The default size of the persistent APD file (the one
    without a numeric suffix) is reduced from 256M to 128M.  As always
    this can be changed with the -sp option.

12. At apformat time, the coverage.ual probe now puts out a new
    (additional) file with suffix .bc which contains individual line
    count information.  This information is for consumption by a
    customer-provided source merging script for selective coverage
    reporting of an integration test and for easier visual
    verification of path coverage.  Contact OC Systems support for
    more information.

13. When the 'apc' command fails due to a non-fatal error, and the
    option -g or -C was specified, the intermediate C file (with suffix
    .apc_c.c) is now preserved as a potential debugging aid.

14. A number of formatting improvements were made in the memstat and 
    java_memstat report files.

Features in RootCause 2.1.3b

 1. Process Intercept support scripts are now provided that
    perform the linking/unlinking based upon an application
    list file.

 2. Enhancements to support Java exceptions on newer JVM versions.

Features in RootCause 2.1.3a

 1. The Trace Events summary tree may now be pruned based upon
    statistic usage ratios between parent and child nodes.

 2. Instrumentation overhead can now be specified and reflected in
    the Trace Events summary tree.

 3. Statistics are now reported for individual nodes of the Trace
    Events summary tree.

 4. Text save/copy operations have been implemented for individual
    nodes of the Trace Events tree.

 5. Function/Method TRIGGERS have been added to the RootCause GUI.
    This mechanism allows tracing to be specified globally and
    controlled dynamically at runtime based upon function/method

 6. Support has been added to allow probes to modify the parameters
    passed to java methods.

 7. stub support has been implemented for java methods.

 8. java run and format -time performance improvements have been

 9. Functions/Methods reported in the Load Shed table are no
    longer reported in the Performance Summary tables.

10. Some trace UAL options have been deprecated.

11. Various improvements have been made to tables in Trace Events.

12. Improvements have been made to the manner in which time is
    computed for recursive function calls.

Features in RootCause 2.1.3

 1. The "rootcause merge" command has been enhanced and a 
    "rootcause copy" command added to better support the use of a general
    Workspace against multiple applications.

 2. The LogTraceback method in TraceBean is now exported to facilitate
    logging throwable exceptions.

 3. The suffix ".xml" is now recognized as an alternative to ".xmj" 
    as a valid suffix for a Java Probe deployment descriptor file.

 4. Support has been added to allow dynamic Workspace selection
    based upon the instrumented application's command line arguments.
    See http://www.ocsystems.com/faqs/rc_aprobe_faq.html#q9.12.

 5. The default limit on number of trace events has been increased to
    1,000,000.  Note that since this is a per-user preference, this
    will only apply to new users or if you change your APROBE_HOME 

 6. Trace logging performance improvements have been implemented for 
    recording machine time.

 7. .zip files discovered in the classpath are now treated just like
    .jar files.

 8. Two new pragmas are available to categorize probe targets. "pragma
    required" (the default if the module is found) identifies a probe
    target that must be present, while "pragma optional" (the default
    if the module is not found) identifies a probe target that need
    not be present. Use of these new pragmas make probes more
    readable, avoid runtime warning messages, and support the ability
    to use parameters from header files.

 9. Custom probes no longer need to explicitly check
    ap_RootCauseTraceIsEnabled to determine if tracing is enabled,
    this is now performed automatically in ap_LogSubprogramEntry/Exit.

10. Performance enhancements have been made to Java method entry/exit logging.

11. Patch count support has been added for the AIX platform, reducing the
    overhead of load-shedding.

12. Enhanced support for per-thread CPU timing is implemented by the
    perf_cpu.ual probe for AIX 5.  The perf_cpu_nothread.ual probe
    preserves the per-process CPU times. 

13. memcheck.apc is now distributed. It is a memory corruption probe
    that uses a "fence" technique to detect buffer under/over runs, 
    as well as multiple deallocates. 

14. A potential thread-safety issue on multi-processor POWER4 systems
    has been addressed.

15. Support for upcoming PowerAda 5.3.1 was added.

Features and Fixes in RootCause 2.1.2

 1. RootCause for Java is fully supported in this version.

 2. The Memory Analysis Probes "memstat" and "java_memstat" have been
    integrated with RootCause.  These probes use statistical sampling
    and analysis to collect memory and Java object usage data about your
    program with low enough overhead to use in real-world testing.
    See http://docs.ocsystems.com/Memstat_Predefined_Probe for a
    complete description.

 3. New performance analysis support has been added to RootCause, with
    which resource data is collected at runtime and then summarized by
    thread and call-site under a new "Event Summary" node.
    The existing trace probe, and new "perf_cpu" and "perf_rusage" probes
    make use of this support to provide much of the power that the
    Aprobe "events.ual" probe provides to command-line users.

 4. New statistical tables for performance analysis have been
    introduced in place of the CALL_COUNTS tables. These provide
    summaries of both call counts and wall-clock times for methods and
    functions, and show additional statistics if provided by predefined
    UALs. This change allows out of the box profiling-style information 
    to be viewed.

 5. Numerous fixes have been made to the handling of PowerAda debug
    information. This version only supports the new PowerAda 5.3 and newer.

 6. gcc (which is the back-end compiler required for compiling PowerAda probes)
    is delivered with RootCause and will be automatically installed and
    configured. This means an external C compiler is no longer required
    for compiling probes.

 7. The time span displayed in the event tree is now clearly displayed.

 8. All event nodes can now be displayed in a text window for better
    integration with standard probes. This also fixes problems with
    the truncation of some useful data such as tracebacks.

 9. The deploy option is no longer licensed separately and customers
    who have any RootCause console license will be able to deploy.

10. Wildcard performance in the trace setup has been improved and a
    number of fixes made to the propagation of the trace / probe

11. Lines are no longer truncated in the source pane.

12. Memstat output in the event viewer has been greatly improved.

13. A liberty.so library is now delivered allowing demangling of gcc 3.x 
    C++ methods. (Note that on AIX gcc 3.x is not specifically supported
    by RootCause though some applications will work correctly).

14. A new cached hashtable facility is now available in the
    runtime. See the ap_Cache... routines in aprobe.h for details.

15. Problems related to enabling and disabling of trace have been

16. A newer version of the xlC demangling library is now used which
    addresses crashes seen with certain C++ symbols.

17. The patch area allocation algorithm has been improved allowing
    more symbols to be instrumented in shared libraries.

18. PowerAda exceptions raised from shared libraries are now handled

19. The memstat and mewatch probes have been adapted to malloc changes
    introduced in AIX 5.2.

Features and Fixes in RootCause 2.1.1/Aprobe 4.3.1

 1. An incorrect ordering in the "View By Time" view in the Trace Display 
    was fixed.

 2. A problem with incorrect nesting of events in the Trace Display was fixed.

 3. The "Remove All Probes" operation in Trace Setup is now much faster.

 4. Installing RootCause from CD no longer requires 'root' privilege 
    (after the CD is mounted).

 5. Very long aprobe command-lines are now accepted.

 6. RootCause now operates by default in a "Developer" mode where 
    the "Deploy" and "Decollect" operations are not supported.  An 
    additional license key is required to enable these operations.

Features and Fixes in Aprobe 4.3 relative to 4.2.5
The significant improvements in Aprobe version 4.3 are listed below.
If you have specific issues you're concerned with, contact OC Systems
directly as described in "Getting Help" above.

1. Support for linking aprobe into an executable [d9606].

   Starting with Aprobe 4.3, you can link the "libdal.so" shared library
   into your AIX executable object module.  This was previously 
   supported only for Solaris, as described in Chapter 4 of the 
   User's Guide.  Use of this feature for AIX is described in 
   "Linking Aprobe Into Your AIX Application" below and
   will be added to the User's Guide in a future version.

2. apcgen "dictionary" mode [f14760]

   A new flag, '-D' has been added to apcgen to list each function
   in the application along with the source file and line number where
   it's declared, if any, in a format that can be postprocessed.
   This is especially useful for Ada applications which have overloaded
   subprograms and it's not clear which overloading is the one you want.
   For PowerAda programs, this works even in the absence of the PowerAda
   library, since source file and line information are stored in the 
   executable itself.

3. apcgen -T, typeof now work for PowerAda programs. [d14706]

   Type names from the PowerAda library are now available in apc code
   and printed by the 'apcgen -T' command.

4. Target expression "-file t.ada" qualifier now works for PowerAda.

   PowerAda now supports the "-file" context specification in APC, 
   in addition to "-unit".

Linking Aprobe Into Your AIX Application
Aprobe includes a special shared library called libdal.so
("DAL" stands for "Dynamic Action Linker"). If you re-link your
application with libdal.so, it will do on startup what aprobe normally
does, and your probes will be in effect. The aprobe options are read
from an APO file as described above.

For example, suppose you're developing a program 'Abcentry' which is
invoked by another program called 'abc', and you want to probe Abcentry
with a UAL file Abcentry.ual.

$ mv Abcentry Abcentry.noprobe
$ cc -o Apcentry Apcentry.c -brtl -ldal -L$APROBE/lib
$ echo "-u Abcentry.ual" > Abcentry.apo

Now you can just run abc as usual, it will invoke Abcentry, and as
long as the Abcentry.apo and Abcentry.ual files are in the current
directory at the time Abcentry is invoked, the probes will be loaded.

The command needed to link with libdal varies with each platform 
and compiler.  

For AIX you need to specify that the runtime linker is used with the -brtl

For AIX C applications:

    cc -o Apcentry Apcentry.c -brtl -ldal -L$APROBE/lib

For PowerAda

   ada -b main ... -i-brtl -i-ldal -i-L$APROBE/lib


  aprojbuild -b main ... -- -i-brtl -i-ldal -i-L$APROBE/lib


  abuild -b main ... -Bi-brtl -Bi-ldal -Bi-L$APROBE/lib

On AIX you can relink an executable if it's not stripped, 
making sure you pass the full set of libraries it was built with.  
For example:

# determine what libraries are referenced by 'hello_mt':
$ dump -H hello_mt | grep "^[1-9]"
1                                    libpthreads.a         shr.o               
2                                    libc_r.a              shr.o

# relink hello_mt with those original libraries plus libdal, plus -brtl:
$ ld -o hello_mt.apx hello_mt -lc_r -lpthreads -brtl -ldal -L$APROBE/lib 

Features and Fixes in Aprobe 4.2.5 relative to Aprobe 3.2.0c

1. Improve shared library patching (patch area acquisition) [d13393]

   Previous versions of AIX Aprobe often encountered problems when
   patching shared libraries.  AIX Aprobe V4 incorporates a new procedure
   for acquiring patch areas for shared libraries so that this is no
   longer an issue.  The use of the 'apcpc' utility for adding patch
   space in local copies of shared libraries is no longer required.

2. shmat no longer used for APD files [d13397]

   AIX Aprobe V3 used the shmat system call to access APD files.  This
   had the potential of interfering with applications which also used
   shmat, and was especially troublesome when those applications used
   shmat without specifying an address to attach to.  AIX Aprobe V4 no
   longer uses shmat so this interference is no longer an issue. In the
   event that there is a conflict with shared memory, the -q mmap=<address>
   option is used instead of -s1.  Contact OC Systems for
   assistance in this case.

3. Less restrictive concurrent-use licensing rules

   Concurrent-licensing problems resulting from processes started from
   different ttys have been addressed by treating all processes run by
   the same user on the same machine as a single user.  Because of this
   change, concurrent-use licenses cannot be issued for both version 3
   and version 4 Aprobe from the same license server.

4. PowerAda Support

   Aprobe and RootCause support source-level probes against programs
   compiled with PowerAda version 5.0 or newer.

5. "Shadow include file" support

   Parameter type information is available for many functions in
   libc.a due to a feature in apc and apcgen that reads this
   information from C header files.

 2. 64-bit applications are not supported by this version. Aprobe64 for AIX