OCS Release Notes Aprobe 4.5.4 Linux32

From OC Systems Wiki!
Revision as of 17:43, 13 September 2021 by Swn (talk | contribs) (Created page with " *READ THIS FIRST* Release Notes for RootCause Version 2.3.4 (Aprobe Vers...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
                         *READ THIS FIRST*
                         Release Notes for
             RootCause Version 2.3.4 (Aprobe Version 4.5.4)
                 for 32-bit Linux/x86 Applications
                            Sep 13, 2021

This file contains brief release notes for RootCause 2.3.4 and Aprobe 4.5.4 for 32-bit Linux x86 platforms. Changes are relative to RootCause version 2.3.3 (Aprobe 4.5.3) released Feb 2021.

Features and Fixes in RootCause 2.3.4/Aprobe 4.5.4 for (32-bit) Linux

(2688) Apc compiler fails to convert class with incomplete base type
(2689) Handling of virtual base classes is problematic
(2691) add probe to log memory for hex dump at format
(2692) create a call path profile probe
(2694) branch coverage warning about unknown opcode
(2695) memcheck does not record C++ allocations
(2699) add AP_MAKE_CPP_STRING to cppstring.h/c
(2704) Linux32 does not handle fixed point type
(2705) Gnat 22/Gcc 10 changed subprogram overload suffixes
(2706) traced inlined code line numbers confusing
(2707) only print Gnat [nnnn] suffix waring in verbose mode
(2708) bad layout for Ada record with dynamic fields
(2709) Gnat 22.0w4 runtime exception signals changed
(2710) Gnat 22.0w4 uses variant/variant_part tags
(2711) bad layout for repc spec'd variant type
(2712) filter compiler generated functiosn in Gnat 22
(2713) remove help output from coverage output
(2715) LINUX 32 DWRF query ap_dwarf_loclist_n fails
(2718) x87 float registers (ST(n)) not correct
(2719) GNAT 22 exception names have @CXXABI suffix

Features in prior releases are listed below (look for "PRIOR").

Getting Help

If you have any questions or problems, please contact OC Systems by e-mail at support@ocsystems.com, or by phone at 703-359-8160.


All OC Systems product documentation is available at http://docs.ocsystems.com.

The RootCause Console 'Help' menu points to the OCS Web site, as well as the full documentation delivered with the product.

The RootCause User Guides are available in PDF format in the files RootCauseJava.pdf and RootCauseCpp.pdf in the product installation directory.


Product Installation and Licensing is described in Chapter 2 of each RootCause User Guide, and in the "install.txt" file located in the root directory of the CD. The installation process has not changed in this update.

After you have successfully installed RootCause, you should work through the Demonstration Program described in Chapter 5 of the RootCause User Guide.

Supported Versions

All support is for 32-bit applications (native or Java only), although they can be running on 64-bit linux kernel.

For 64-bit applications, you must download and install the corresponding 64-bit version.

We support and have tested with Redhat versions 9.0, Red Hat Enterprise 2, 3, 4, 5, 6, 7; and corresponding Centos and Fedora Core versions. We fully support C and C++ programs compiled with GCC version 3.x and 4.x compilers using -g. We support Java programs compiled with Sun Java 1.4 or newer.

This release fully supports 32-bit programs compiled with public GNAT 4.4.7 and the latest versions of GNATPro GNAT (currently 7.4.1 and 17.0 wavefronts).


This release is source-code compatible with previous versions, so you shouldn't need to change any APC files. However, this version is NOT binary-compatible, with respect to UAL files. That means you must recompile your probes from their APC source.


The libiberty.so shared library is delivered under the terms of the Lesser GPL. A description of this, and a copy of the license, can be found in README.libiberty.


This version of RootCause requires libelf.so.1 to be in your library path. If you don't have this contact support@ocsystems.com and we'll help you find one for your linux version.

Dynamically-loaded (dlopen) libraries

Tracing a "dynamic module" C/C++ library causes it to be statically loaded at program start (via aprobe using -dll command). This means that one must be sure that the library specified in the RootCause workspace must be the exact same one which the program would normally load. Furthermore, this won't work at all for libraries whose initialization depends on program state. Any -dll libraries that cause libc.so to be loaded at a different address than defined in the program header will cause problems.

Setuid Applications

Applications which are owned by root and have the "setuid" bit set require special consideration. Contact support@ocsystems.com for assistance.


Java probes written completely differently than native (C/C++/Ada) probes. See the User's Guide and/or contact support@ocsystems.com about probing Java.

Logging data over NFS

Aprobe's log mechanism uses memory-mapped APD files to achieve its excellent performance. When the APD files are located on an NFS-mounted disk, corruption or loss of data logged to multiple APD ring files may occur. This has always been the case, and Aprobe and RootCause documentation encourages users to log to files on local (or at least non-NFS-mounted) disks. However, it can be very confusing when it does happen and you don't realize it, so we introduced a warning (only with aprobe -v) and a mechanism to avoid the problem. The mechanism is enabled by exporting the environment variable AP_NFS_TMP=1 before running aprobe. Aprobe will recognize when NFS logging is taking place, and instead log under /tmp. When program execution is complete, the apd files are copied to the final destination. Until this copy occurs, the APD file is instead a text file that explains what's going on. If you would like more information, or have problems using this feature, please contact support@ocsystems.com.


You may be interested in these features/changes introduced in prior releases of RootCause and Aprobe.

Features and Fixes in RootCause 2.3.3/Aprobe 4.5.3 for (32-bit) Linux

(2678) Create probe to track (large) memory operations
(2679) hmtb.ual doesn't show tracebacks
(2681) Allow more flexible call patches (on Linux).
(2686) Aprobe fails on RHEL8 program wiht dlopen() exceptions

Features and Fixes in RootCause 2.3.2/Aprobe 4.5.2 for (32-bit) Linux

(2637) Coverage snapshot missing for forked pids
(2640) statprof report improvements
(2642) Gnat20/Gcc8 optimized code hard to probe
(2645) Create apcgen -U type dictionary.
(2646) union field type not found in apc.
(2647) hmp report label MaxOutstanding not clear
(2648) Apcentry can't handle struct with field of stub type
(2649) create HMTB - heap memory traceback probe
(2650) Struct field references don't compile in apc
(2651) prototype using a overloaded function selector
(2654) ap_ThrowGccString does not work with -q min_heap

Features and Fixes in RootCause 2.3.1/Aprobe 4.5.1 for (32-bit) Linux

(2592) COVERFILE should exclude (more) compiler generated func
(2604) enhancements to memcheck: -S -e -E options
(2610) Add pragma trap_patch to allow a trap patch on PPC
(2613) provide mechanism to register dll/so with Aprobe
(2614) Linux compiler-generated member names with special char
(2615) core dump using ap_GnatRaiseException
(2616) small return type does not set entire reg
(2617) branch coverage report should show which tests take branch
(2618) Linux use of ap_StringValue() in on_entry fails to compile
(2619) Linux/AIX ap_StubRoutine on_line/offset does not work
(2620) memcheck should support resetting freed block values
(2621) Add indication of inlined function to apcgen -D
(2622) ap_ThrowGccException does not work on Linux
(2623) Gcc optimized hot/cold funcitons have no debug info
(2626) apc compile error: Expecting an operand of a numeric type RHEL8
(2628) track taken test names in brcov
(2629) stdlib.h compile error using Apcentry on REHL8
(2630) apc parse error on slice range with Gcc 8
(2631) fail apc compile of __alignof__ keyword on RHEL8
(2632) Aprobe fails at startup on RHEL8
(2633) Gnat exceptions not working with exit probes
(2634) ap_RaiseGnatException() not working on_line
(2635) ap_ThrowGccException() not working on_line

Features and Fixes in RootCause 2.3.0/Aprobe 4.5.0 for (32-bit) Linux

(2552) statprof -a timings off due to bogus extra entry
(2557) statprof -c fails on linux64
(2558) COVERFILE should exclude compiler generated functions
(2559) include date in statprof timestamp
(2560) APO file doesn't handle single quote for options
(2561) coverage does not do branches for some functions
(2562) Linux Gcc try harder to find exception name/message
(2563) Gnat/Gcc parameters not referenced correctly when -O
(2564) struct with ap_TimeT fields get false unions
(2567) x86/64 disassemlber sets float mem ins class OTHER
(2568) Linux/Gnat Apcentry doesn't allow target type expr outside probe
(2569) Linux/Gnat dynamically-sized struct fields not generated
(2570) apcgen core dump processing optimized Gnat code
(2572) add heap memory profile (hmp) probe
(2574) Handle Gnat optimized symbol names
(2575) Gnat optimized code has formals in lexical block

Features and Fixes in RootCause 2.2.9/Aprobe 4.4.9 for (32-bit) Linux

(2510) brcov.cfg missing from delivery
(2511) add ability to prune ADI files 
(2512) Apc generates bad layout for variant record 
(2513) apc core dump when emiting warning message
(2515) support GNAT Pro 19.0
(2516) apc can't compile gcc std::tuple with formal parm __ele
(2517) merge brcov into coverage and extras
(2518) problem with GNAT exception propagation
(2520) Linux gcc problem reading static member variable
(2521) add early Aprobe debug
(2522) fail to handle gcc const variables
(2523) DWARF2 has no location for extern var
(2524) failure to find g++ sepcial symbols
(2525) x86 support for XMM registers
(2527) default name matching for .adi files not convenient
(2528) apbc should process similar .brc files
(2529) memleak and memwatch should have delayed start
(2531) apcgen reads all debug even if -f used
(2533) coverage.ual core dumps if -if used
(2535) apbc doesn't handle similar .brc files with .eab
(2537) g++ const value uses 4 byte block
(2539) provide option to set statprof interval
(2540) Upgrades to apcgp. 
(2541) abrmerge should warn if source file not found
(2545) add statprof option to profile address range
(2546) problem probing __x86.get_pc_thunk.XX() on Linux
(2547) gcc 7.3.1 C++ symbols not demangled
(2549) Linux params/locals not correct on_line

Features and Fixes in RootCause 2.2.8/Aprobe 4.4.8 for (32-bit) Linux

(1197) Linux: exit() from thread doesn't trigger program on_exit
(2490) provide axdln for Linux
(2492) axdln doesn't tolerate # lines
(2493) apc rejects -unit name with mixed case
(2494) apc should prefer local symbol if file scope given
(2495) Further branch coverage improvements
(2496) GNAT Linux reference to inout param in stubbed function wrong
(2497) apcgen -T and -M don't work
(2499) apmkadi on Linux checksum relies on stripped item
(2501) core dump on stripped Linux executable
(2505) create a name demangling tool
(2507) compile errors in APC compile for array struct members

Features and Fixes in RootCause 2.2.7/Aprobe 4.4.7 for (32-bit) Linux

(2449) brcov doesn't handle lines from multiple files
(2450) cannot log GNAT static-sized record subtype
(2451) support enable/disable calls and apdemand in memstat
(2453) brcov improvements
(2454) COVERFILE files cause core dump when inserting multiple files
(2457) apbc.pl doesn't correctly mark some lines
(2460) apcgen warnings fo C++ symbol _Z
(2461) apcgen -La reads all debug info
(2462) apbc/bc.pl does not handle unprobed lines in .bc
(2463) coverage does not handle fork() well
(2465) Trap patch on line at end of function not made
(2469) apc problems with wavefront 18.0w 20170628
(2471) GNAT 18.0 w20170601: main-nested subp debug info not fo
(2472) typo in error message
(2474) Aprobe setup script must check for all of Aprobe
(2476) Upgrade i386 disassembly
(2477) apc generates incorrect layout for repspec'd variant record
(2480) no line information in Gnat shared library
(2481) Linux libdal.so does not support preinit
(2484) -qmin_heap breaks memstat traceback table
(2485) apc defines unknown types for type nested in subp decl

Features and Fixes in RootCause 2.2.6d1/Aprobe 4.4.6d1 for (32-bit) Linux

(2464) GNAT exception reraise changed for wavefront 18.0w 2017
(2468) aprobe setup script doesn't handle blanks in PATH

Features and Fixes in RootCause 2.2.6d/Aprobe 4.4.6d for (32-bit) Linux

(2373) wrong filename scope for GNAT symbols
(2374) $APROBE/setup should update library path
(2375) Warn if probing system calls
(2376) log fixed-length string return value fails
(2378) partial coverage percentages not obvious
(2379) Linux target expression problems
(2382) Linux/Gnat exception reason string not right
(2383) Add support to raise C++ exception from probe
(2384) Linux/Gnat position of fixed-length string params wrong
(2387) emit DWARF warning messages in verbose mode
(2388) atcmerge should print Aprobe version
(2390) support gnatpro 18.0w for APTs
(2393) support apcgen -D for big Linux executables
(2394) support wildcard in apcgen -p argument
(2395) support logging local/global variables in apcgen
(2396) make ExpandDottedLines the default for coverage
(2397) support COVERFILE 'sourcefilename' in coverage.cfg
(2399) fix logging Linux return structs
(2401) Coverage should report reasons functions not probed
(2402) Gnat 18.0w DWARF subtypes before base type
(2404} Warn about mismatched function version in coverage
(2405) atcmerge should allow merging diff func versions
(2406) apbc should warn if no source files found/given
(2407) Warn if Gnat [nnn] suffix not needed
(2409) atcmerge should produce .bc file as an option
(2411) Allow aliases of module names (libc.so.6)
(2413) Aprobe Linux32 stat/fstat/lstat on 64-bit file system
(2415) Coverage should report maximum offset count for line count
(2423) Add source file matching option to apcgen
(2424) apcgen generated incorrect scope for func-local var
(2425) apc can't reference Gnat out params when optimized
(2426) Problem stubbing abort() with coverage.ual
(2430) provide access to STL exception message
(2431) exceptions.ual should provide Ada/C++ exception message string
(2432) move DWARF warnings to verbose mode
(2435) Linux/Gnat tag field is hidden when logging
(2436) Linux on_exit probe breaks cross-lang exception raise.
(2441) memstat statistics are displayed incorrectly
(2443) provide traceback blacklist address support
(2445) Linux/Gcc handle cxx11 tag when demangling names
(2455) apc generated bad code for Gnat in array param ref

Features and Fixes in RootCause 2.2.6c/Aprobe 4.4.6c for (32-bit) Linux

(2370) wrong symbol filename used due to #include
(2371) Linux aprobe fails to demangle 'gnat170w9' symbols

Features and Fixes in RootCause 2.2.6b/Aprobe 4.4.6b for (32-bit) Linux

(2280) Linux Aprobe problem with LD_PRELOAD in forked process
(2300) Linux/GNAT upscope references not supported
(2318) bc support for linux
(2319) Linux/Gnat exception reason string not reported.
(2321) sigsegv.ual causes core dump on Linux386
(2322) Linux/Gnat Interfaces.C.Long_Float not handled
(2323) Linux/Gnat records with dynamic sized components layout
(2326) Linux apc fails when probes in same file
(2332) Continuation of i2329 Gnat tracebacks
(2333) Linux (F) 'thread stacks were overlapped'
(2334) coverage.ual should do final snapshot on abort
(2335) Linux coverage gets bad line info, causes sigsegv
(2337) C++ const int declarations not handled
(2338) ap_RaiseGnatException broken for Gnat shared library
(2340) implement ap_UnprobeThread to avoid GNAT stackwalk core
(2341) C++ exceptions not translating correctly
(2342) GNAT abort task causes core dump unwinding
(2346) Switch to pthread TLS support for thread tables
(2348) Linux/GNAT: on_line probe gets wrong param value in nes
(2349) use lower case exception name for GnatRaiseExceptionByN
(2350) support ap_GnatRaiseException in on_entry and on_line

Features and Fixes in RootCause 2.2.6a/Aprobe 4.4.6a for (32-bit) Linux

(2324) Aprobe traceback skips a frame when exit probe present
(2329) Linux/Gnat cannot instrument lines with extended float instructions

Features and Fixes in RootCause 2.2.6/Aprobe 4.4.6 for (32-bit) Linux

(2090) Linux/GNAT: update out parameter on_exit wrong
(2243) Linux apcgen, apc should work with g++/gnat .o files
(2259) GNAT/Linux core dump in __gnat_backtrace
(2267) bad reference from probe for GNAT out parameter
(2272) linux problems with ap_CallerLocation
(2277) Linux atcmerge core dump
(2278) Linux/GNAT: problems logging ada.bounded_string component
(2279) GNAT/Linux apcgen wrong for param after string param
(2282) Gnat/Linux record layout problem
(2283) deliver libdal.so.stub for Linux
(2285) Gnat/Linux support for backtrace(3)
(2286) Linux/Gnat apcgen assert on char16_t predef type
(2287) Linux/g++ DW_DLE_ATTR_FORM_BAD on huge struct
(2288) Linux/g++ exception class ctor debug info not found
(2289) Linux/Gnat Ada out param with cast wrong
(2290) apc: object location cannot be determined
(2293) Gnat/Linux access to in param on_exit
(2295) Gnat/Linux array slice lower bound not static
(2297) apc c constant array treated as Ada
(2298) gnat generates REP component for partial rep spec
(2299) Linux apcgen prints wrong source file name
(2307) Linux/Gnat filenames wrong for template methods
(2310) use MAP_ANONYMOUS on linux
(2311) Linux/Gnat problems with exit probes, exceptions, Ada-C++
(2317) deliver call graph probe apcgp.ual
(2318) bc support for Linux

RootCause 2.2.5d/Aprobe 4.4.5d for (32-bit) Linux

1. (1361) add changes, comments to coverage.2/bc.pl
2. (1477) coverage sometimes fails to report missed lines
3. (2203) coverage.cfg should allow inline comments
4. (2239) Linux/386 bad messages for unkown ins type
5. (2240) no coverage snapshot on SIGINT or exception
6. (2241) atcmerge fails merging .tc files from multiple apps
7. (2242) apcgen -f doesn't handle include files
8. (2244) apcgen misses C++ members fields in big class
9. (2246) failure for slice of array'access

10. (2248) core dump on ap_CallerLocation 11. (2249) optionally reset counts between coverage snapshots 12. (2250) each coverage snapshot should write separate .tc and .bc 13. (2253) coverage summary report line should put symbol last 14. (2254) coverage CommaOutputFile gets duplicate content 15. (2256) improvements to coverage reports 16. (2261) add experimental expand-dotted-lines tool 17. (2262) support .bc files on Linux 18. (2263) no APC type created for GNAT rep spec'd field 19. (2264) handle private incomplete types in GNAT 20. (2269) enhance statprof to profile enitre address space 21. (2270) report all default options in coverage cfg file

RootCause 2.2.5c/Aprobe 4.4.5c

1. (2186) GNAT array base address computation incorrect
2. (2189) GNAT exceptions not intercepted
3. (2198) GNATPro/Linux: core dumps reading DWARF
4. (2199) core dumps instrumenting Firefox
5. (2205) problem parsing gcc 5.x attribute list
6. (2207) ap_DoCoverageSnapshot() doesn't work
7. (2213) Linux/GNAT CompilerKind wrong if package 'sec'
8. (2215) Linux/Aprobe/GNAT core dump reading line information
9. (2224) Linux assert failure in ap_CallerLocation

10. (2237) apcgen doesn't recognize C++ built w/GNAT

Features and Fixes in RootCause 2.2.5b/Aprobe 4.4.5b for (32-bit) Linux

1. (2138) problems building on centos7
2. (2139) lines not logged if duplicate timestamp
3. (2141) apc parse error parsing complex gnu C type decl
4. (2144) gcc4.8 requires dwarf4 support
5. (2160) fix memwatch, memstat examples, output
6. (2164) java aborts under RC trace

Features and Fixes in RootCause 2.2.5a/Aprobe 4.4.5a for (32-bit) Linux

1. (2117) show instantiation line in apc/apcgen comments

Features and Fixes in RootCause 2.2.5/Aprobe 4.4.5 for (32-bit) Linux

1. (1182) on_line(last) doesn't work for generic instance
2. (1921) avoid NFS apd logging
3. (2051) Linux: coverage breaks I/O app
4. (2069) unhelpful message for $a[i, j]
5. (2085) support public gnat on linux

Features and Fixes in RootCause 2.2.4/Aprobe 4.4.4 for (32-bit) Linux

1. (1894) rhel6: sometimes can't instrument on_line(first)
2. (1996) problems with trace.ual in PA/ada2cpp/g++ Linux applications
3. (2002) PA eqOp symbol demangled wrong by RC223a

Features and Fixes in RootCause 2.2.3a for (32-bit) Linux

1. (1962) (OCS) don't retry failed demangling
2. (1964) (OCS) provide libaprobe.adi

Features and Fixes in RootCause 2.2.3 for (32-bit) Linux

1. (1865) No module found for pa4_get_reraise_ptr(0)
2. (1880) apcgen: '(E) ap_dwarf_loclist_n: DW_DLE_ATTR_FORM_BAD'
3. (1885) PROGRAM_ERROR from apc refing C file in Linux PA exe.
4. (1892) local data addressed wrong on rhel6/gcc4
5. (1842) Linux mismatched ADI files not flagged
6. (1884) function inside probe thread hangs
7. (1896) apc -v shouldn't do gcc -v
8. (1897) rhel6, handle non-standard frame base reg
9. (1899) rhel6, Apcentry can't demangle g++ names

10. (1900) gcc4.4, g++ file-static debug info not found 11. (1911) typeof doesn't find C types in Ada binary 12. (1912) redefinition of typedef .. PtrTo_void 13. (1930) provide C++ string helper UAL

All of the above fixes also apply to the 64-bit version of the product.

Fixes 1..4 were delivered as patch 4.4.2i-p1 on November 16, 2011.

Issue 1930 introduces "cppstring.ual" which provides ap_GetCppString() and ap_SetCppString() functions for use in your probe. Its usage is illustrated in $APROBE/examples/predefined_probes/cppstring/.

Features and Fixes in RootCause 2.2.2i

1. (1020) sigsegv stack trace incomplete
2. (1802) port sigsegv.apc to 64-bit Linux
3. (1804) support universal_traceback in PowerAda/Linux

Features and Fixes in RootCause 2.2.2h

1. (1724) ap_DoPeriodically() should be cancellable
2. (1770) support gcc4 optimized code
3. (1773) handle some linux386 struct return logic (tzsrgsc1)

Features and Fixes in RootCause 2.2.2g

1. (1730) fix Linux Aprobe/Java failure for JRE 1.6.0_18
2. (1738) fix apc parse error for c99 'restrict' keyword
3. (1756) remove ksh requirement 

Features and Fixes in RootCause 2.2.2e

1. (1207) apformat should remember ADI files (see NOTE 1)
2. (1386) (OCS) fix string leak in CopyNode
3. (1434) (OCS) provide workaround for bad type layout (see NOTE 2)
4. (1436) (OCS) refactor type replacement implementation
5. (1441) (OCS) implement 'formatof' to manipulate apc print routines (NOTE 2)
6. (1442) different apc results for upper, lower case
7. (1485) formatof + typeof don't work together (NOTE 2)
8. (1497) RC/Linux: g++ 3.4 trace setup problems
9. (1551) Provide memleak ual for Aprobe 

10. (1561) RCLinux: aprobe DOA on FC6 11. (1570) Linux: make scripts more linux-friendly 12. (1582) JVMTI: aprobe.jar has classes not in com.ocsystems 13. (1600) Linux: aprobe runtime can't find lines in .so file 14. (1602) Update linux examples 15. (1621) ap_DoPeriodically incorrectly uses signal on threaded app 16. (1624) Linux: FC9 parse error on __signed__ 17. (1625) Linux: gcc4 line numbers all wrong 18. (1626) Linux: gcc4 exceptions not supported 19. (1631) Linux: probes on libc from .h files don't work 20. (1632) Linux: profile.ual's benchmark broken 21. (1633) Linux: RC trace setup has no debug info due to libdwarf 22. (1635) Linux: do 'set +u' in setup scripts to avoid breakage 23. (1636) Linux: problems with log_env probe

 NOTE 1: apformat now "remembers" ADI files (i1207)
 By customer request, aprobe logs the paths of ADI files that were
 loaded at runtime, and apformat attempts to load these again at
 format-time.  apformat will not fail if they are absent, and apformat
 can be forced not to load them by renaming the .adi file itself, or
 specifying -z on the apformat command line and explicitly naming UALs
 that are wanted.  Also, apformat -u info -p "-u" ... now lists ADI
 files applied at runtime.
 NOTE 2: APC auto-formatting extensions added (i1434, i1436, i1441, i1485)
  This version introduces two extensions to the APC language to allow
  user-specified replacement of default apc behavior.
  ** typeof Declaration **
  The first extension is a new use of the keyword 'typeof' to designate
  a locally defined APC type to be used in place of the type that the
  apc compiler would generate to overlay the in-memory image of a
  target program's type.  For example:
    typedef struct
       int word0, word1;
    } my_time_type;
    typeof $time_type  my_time_type;
  Then wherever an object or field of type $time_type was logged, it
  would be interpreted as type my_time_type.
  Why would you do this?  Mostly to work around bugs or limitations in
  the APC compiler.  The above is a real-world case where time_type was
  defined as a 'double' but the 8-byte alignment imposed by the apc
  backend compiler didn't match the 4-byte alignment used by the target
  compiler.  By replacing the 'double' with a struct consisting of 2
  ints, the alignment restriction was removed and everything worked.
  The onus is entirely on the user to define a compatible type; there
  is no checking for compatibility.  The original type layout must be
  determined by looking at the C code generated by the 'apc' compiler:
    apc -g t.apc -x t.exe  # creates t.apc_c.c file
  This can be further cleaned up and made readable, for example:
    cat t.apc_c.c | \
      grep -v "^#line" | \
      sed "s/^ *//" | grep -v "^$" | \
      indent -st -sob -dj -i3 -bl | expand -3
  It can still be complicated, so generally the replacement type will
  be determined in cooperation with OC Systems support.  This can then
  be defined in an include file that users can include into their
  probes as needed.
  ** formatof declaration **
  The typeof extension described above led to the need for a
  corresponding formatting extension, so that the replacement type
  could be printed as if it were the original, or in any other way the
  user desires:
     formatof type_name print_proc_name;
  This declaration says: when generating the automatic formatting code,
  call the already-defined function 'print_proc_name' to print any 
  object or field of type 'type_name'.   A "print proc" always has the
     void print_proc_name(int Indent, ap_NameT Prompt, type_name *Arg);
  For example, see the functions ap_Print_* in aprobe.h.  So, to use
  the formatof declaration to print "my_time_type" as a double value,
  you would do something like:
     static void 
     Print_my_time_type(int Indent, ap_NameT Prompt, my_time_type *Arg)
        double d;
        memcpy(&d, Arg, sizeof d);
        ap_Print_double(Indent, Prompt, d);
     formatof my_time_type Print_my_time_type;
  As with typeof, there is another form of 'formatof' which gives
  the existing print procedure name for a given type or expression:
     formatof( type_name )
  So you could replace the call to ap_Print_double above with:
     formatof(double) (Indent, Prompt, d);
  Yes, it looks strange, but it works.
  As with 'typeof', it's up to the user to ensure that the print
  routine is declared with the right prototype and does something
  sensible.  As with format routines the Arg parameter is passed by
  reference as in the example above.
  The major limitation we know of right now is that the 'formatof'
  mechanism doesn't work for array types, even fixed-sized ones.  
  This is because of the special handling of such types to capture the 
  size of an array as a "slice".  
  Aprobe historians may know that 'formatof' was originally defined
  as part of the language, for actually defining format routines.
  While the syntax was recognized, it was never implemented because it
  didn't fit the general usage of automatic formatting.  This seemed
  like a good way to capture the original intent.

Features in RootCause 2.2.2

1. RootCause now supports 32-bit apps on 64-bit linux. (i735)
   64-bit applications are still not supported.
2. Parameters of type 'long double' are now supported. (i787)
3. Numerous improvements to the handling of ADI files. (i829, i844, i963, 
4. Reduced runtime memory usage, including an option -qmin_heap which
   discards symbol table after instrumentation. (i879, i885, others)
   (See NOTE 1 below.)
5. 'Find Function in Trace Events' now works from Event Summary Tree. (i902)
6. The mechanism for disabling/enabling probes has been overhauled
   with more reliable mechanisms (i948). (See NOTE 2 below.)
7. RootCause 'Build' operation now invokes prebuild.ksh, postbuild.ksh in 
   workspace if present, writing output to Message window. (i1014)
8. Extended functionality of APROBE_DUMP_CORE_IF_ERROR. (i1018)
   (See NOTE 3 below.)
9. Fixed crash on thread data initialization. (i1049)

10. Fixed Java 1.5 class verification error. (i1155)

11. Added 'apformat -f', analogous to 'tail -f', to format apd file(s) as they

   are written.  (See NOTE 4 below.)

12. Fixed problem with JVM and java -server invocation. (i1191)

13. Add support of IBM JVM on Linux i386. (i1204)

14. Make all -v, -!, ap_Debug() and ap_DumpMemory() output go to stderr.


15. Fixed problems with Linux runtime module location, size on RHEL4.

   (i1211, i1280)

16. Make ap_DoPeriodically() use threads when available on Linux. (i1231)

17. Removes spurious warnings when using gcc 4.x.

18. Avoid optimization problems with params on_exit by using saved on_entry

   params unless overridden with 'apc -y'. (i1289) 

19. Use libelf.so rather than libelf.a. (i1298)

20. Fixed a race condition in ap_HashTableDelete. (i1303)

21. Re-enabled support for DisplayReport SubtotalReport in events.apc. (i1310)

Features in RootCause 2.1.4b

1. RedHat Fedora Core 3 and Enterprise Linux 4 (kernel 2.6.x) are now 
2. The function ap_GetHostId() was changed to return the same value
   as the "hostid" command instead of the license manager's host ID, 
   which is now provided by the new function ap_HostLicenseId().  
   The function ap_GetHostName() which returns the same string as 
   "hostname". (i647)
3. The memwatch, memstat java_memstat probes now support "REMOVE"
   filters which identify allocations that should not be reported.  
   This are documented in the sample configuration files for these
   probes provided in $APROBE/probes. (i650)
4. The ADI (Aprobe Debug Information) support has been improved in
   several ways: aprobe will look for the default-named adi file in
   the same directory as the application and will use it if it's
   checksum matches; apmkadi -v reports more information; aprobe -v
   will report whether ADI file was used and if not, why not (e.g.,
   checksum mismatch).  (i680)

Features in RootCause 2.1.4a

1. The management of APD rings in the presence of frequent process creation
   was made more robust (i409).
2. apinfo -l now prints the source file name associated with source
   lines (i538).
3. Several improvements were made to the 'statprof' probe (i579, i585).
4. PowerAda shared libraries are now supported by the RootCause 
   Trace Setup dialog (i581).

Features in RootCause 2.1.4

1. Programs compiled with GCC version 3.x compilers with the "-g" flag
   (which generates DWARF debug information) are now fully supported.
2. Several critical defects in RootCause deploy functionality were 
3. The statprof.apc probe was ported to linux.  This probe enables
   the kernel services for performing statistical profiling on a
   given address space range.
4. The events probe was enhanced to export JNI calls for all
   functions defined in events.h.
5. Improvements were made in the way dot-indicators are propagated
   in the GUI's Trace Setup window.
6. A problem where Aprobe disturbs the global "errno" value was been
7. A new option (-r) has been implemented for the statprof custom
   probe that resets the counts to zero for each snapshot. This will
   result in profiling results for each interval between snapshots
   which are independent of each other, rather that the default which
   is cumulative.
8. Java 1.5 is now supported.
9. Several corrections were made to the custom memcheck probe.

10. A custom probe is provided to allow instrumentation of the

   de-facto standard for Java logging, log4j. This probe provides the
   ability to watch the configuration file to look for changes.

11. Handling of very large Java methods has been improved.

12. A new function is added to aprobe.h:

      extern void ap_HeapStats(ap_Uint64   *TotalBytesAllocated,
                               ap_Uint64   *TotalBytesFreed);
   This may be called any time in a probe to determine the amount of
   memory allocated by Aprobe itself, separate from the application
   being probed.  When running aprobe with the -v flag, these
   statistics are printed at various points during program startup, for
      Aprobe: Initialized thread stack
      Aprobe: Heap memory stats
              Allocated bytes: 26016
              Freed bytes: 0

13. The Aprobe run-time system and probes, as well as any user probes

   that use ap_Malloc(), now allocate heap memory from shared memory
   allocated using mmap() rather than from the program's memory space
   using malloc().  This allows Aprobe to handle applications with huge
   symbol tables and memory needs.
   The aprobe configuration parameter "-qmmap=<address>" has been
   extended to apply to all uses of mmap including this heap memory
   allocation.  Hence the specification, "-qmmap=0x20000000" would
   indicate that the first mmap operation would begin at that address
   and each subsequent mmap operation would request the next
   available page after that, whether used for APD files, aprobe
   heap, or another purpose.
   In order to allow all aprobe shared memory usage to fit into a
   single segment by default, the default size of a single APD file was
   reduced from 256M to 64M.  As always, this can be changed with the
   -s option and multiple APD files can be specified with the -s
   option.  The default size of the persistent APD file (the one
   without a numeric suffix) is reduced from 256M to 128M.  As always
   this can be changed with the -sp option.

14. When the 'apc' command fails due to a non-fatal error, and the

   option -g or -C was specified, the intermediate C file (with suffix
   .apc_c.c) is now preserved as a potential debugging aid.

15. A number of formatting improvements were made in the memstat and

   java_memstat report files.

Features in RootCause 2.1.3a

1. The Trace Events summary tree may now be pruned based upon
   statistic usage ratios between parent and child nodes.
2. Instrumentation overhead can now be specified and reflected in
   the Trace Events summary tree.
3. Statistics are now reported for individual nodes of the Trace
   Events summary tree.
4. Text save/copy operations have been implemented for individual
   nodes of the Trace Events tree.
5. Function/Method TRIGGERS have been added to the RootCause GUI.
   This mechanism allows tracing to be specified globally and
   controlled dynamically at runtime based upon function/method
6. Support has been added to allow probes to modify the parameters
   passed to java methods.
7. stub support has been implemented for java methods.
8. java run and format -time performance improvements have been
9. Functions/Methods reported in the Load Shed table are no
   longer reported in the Performance Summary tables.

10. Some trace UAL options have been deprecated.

11. Various improvements have been made to tables in Trace Events.

12. Improvements have been made to the manner in which time is

   computed for recursive function calls.

13. RootCause now ships with version 1.4.2 of the java JRE.

Features in RootCause 2.1.3

1. The "rootcause merge" command has been enhanced and a 
   "rootcause copy" command added to better support the use of a general
   Workspace against multiple applications.
2. The LogTraceback method in TraceBean is now exported to facilitate
   logging throwable exceptions.
3. The suffix ".xml" is now recognized as an alternative to ".xmj" 
   as a valid suffix for a Java Probe deployment descriptor file.
4. Support has been added to allow dynamic Workspace selection
   based upon the instrumented application's command line arguments.
   See http://www.ocsystems.com/faqs/rc_aprobe_faq.html#q9.12.
5. The default limit on number of trace events has been increased to
   1,000,000.  Note that since this is a per-user preference, this
   will only apply to new users or if you change your APROBE_HOME 
6. Trace logging performance improvements have been implemented for 
   recording machine time.
7. .zip files discovered in the classpath are now treated just like
   .jar files.
8. Two new pragmas are available to categorize probe targets. "pragma
   required" (the default if the module is found) identifies a probe
   target that must be present, while "pragma optional" (the default
   if the module is not found) identifies a probe target that need
   not be present. Use of these new pragmas make probes more
   readable, avoid runtime warning messages, and support the ability
   to use parameters from header files.
9. Custom probes no longer need to explicitly check
   ap_RootCauseTraceIsEnabled to determine if tracing is enabled,
   this is now performed automatically in ap_LogSubprogramEntry/Exit.

10. Performance enhancements have been made to java method entry/exit logging.

11. memcheck.apc is now distributed. It is a memory corruption probe

   that uses a "fence" technique to detect buffer under/over runs, 
   as well as multiple deallocates.

12. Support for upcoming PowerAda 5.3.1 was added.

Features and Fixes in RootCause 2.1.2

1. The -jar option is now supported for Java. In order for this to work, 
   the .jar file must be used to initialize the workspace (normally
   this will be automatic).  The previous workaround, changing this to
   specify the .jar file on the classpath and explicitly providing the
   main class, may be removed.
2. Exceptions raised by C++ code compiled with gcc 3.x are now supported.
3. Numerous fixes have been made to the handling of PowerAda debug
   information. This version only supports the new PowerAda 5.3 and
4. The time span displayed in the event tree is now clearly displayed.
5. All event nodes can now be displayed in a text window for better
   integration with standard probes. This also fixes problems with
   the truncation of some useful data such as tracebacks.
6. New statistical tables for performance analysis have been introduced,
   in place of the CALL_COUNTS tables. These provide summaries of both
   call counts and wall-clock times for methods and functions, and
   show additional statistics if provided by predefined UALs. This
   change allows out of the box profiling-style information to be viewed.
7. A new option to log parameters for multiple Java methods has been
   introduced in the trace setup window.
8. The deploy option is no longer licensed separately and customers who have
   any RootCause console license will be able to deploy.
9. RootCause now correctly handles java (1.4) command lines where the JVM
   kind (server, client, etc.) is not the first parameter specified.

10. Wildcard performance in the trace setup has been improved and a number of

   fixes made to the propagation of the trace / probe markers.

11. Lines are no longer truncated in the source pane.

12. Memstat and Java memstat output in the event viewer has been greatly


13. A liberty.so library is now delivered allowing demangling of gcc 3.x

   C++ methods.

14. A new cached hashtable facility is now available in the runtime. See the

   ap_Cache... routines in aprobe.h for details.

15. The Memstat probes now correctly analyze the data when long runs are used

   in combination with a format start time option.

16. The Java memstat probe will disable the -parallel GC option which is not

   supported by the JVMPI interface.

17. The perf_cpu and perf_rusage probes are now included in the agent


18. Problems related to enabling and disabling of trace have been addressed.

19. Formatting from the RootCause GUI has been fixed on RedHat 9. There appears

   to be a bug in the system call but this has been worked around.

20. A problem with the thread-stack handling code on Linux has been fixed.

21. Problems with attempting to demangle Fortran symbols as if they were Ada

   symbols have been fixed.

22. The licensing software has changed and now uses the same version across

   all platforms. The upshot of this for Linux is that new license keys are

Features and Fixes in RootCause 2.1.1

1. You can now print graphs produced by the memstat and java_memstat
2. A workaround for a JVMPI bug causing OutOfMemory errors was added 
   to java_memstat.
3. Empty jar files and empty methods are now tolerated by RootCause.
4. You can now directly add a classpath entry via "Find Class..." 
   in the Trace Display.
5. An incorrect ordering in the "View By Time" view in the Trace Display 
   was fixed.
6. A problem with incorrect nesting of events in the Trace Display was fixed.
7. The "Remove All Probes" operation in Trace Setup is now much faster.
8. Installing RootCause from CD no longer requires 'root' privilege 
   (after the CD is mounted).
9. Very long aprobe command-lines (e.g., due to long classpath)
   are now accepted.

10. An off-by-one problem with line numbers in tracebacks was fixed.

11. Several problems mapping PowerAda symbols to the corresponding debug

   information in the PowerAda library have been fixed.

12. RootCause now operates by default in a "Developer" mode where

   the "Deploy" and "Decollect" operations are not supported.  An 
   additional license key is required to enable these operations.

Features and Fixes in RootCause 2.1.0

1. The Memory Analysis Probes "memstat" and "java_memstat" have been 
   integrated with RootCause.  These probes use statistical sampling
   and analysis to collect memory and Java object usage data about your
   program with low enough overhead to use in real-world testing.
   See http://www.ocsystems.com/user_guide/probes/memstat.html for a 
   complete description.
2. New performance analysis support has been added to RootCause, with
   which resource data is collected at runtime and then summarized by
   thread and call-site under a new "Event Summary" node. 
   The existing trace probe, and new "perf_cpu" and "perf_rusage" probes 
   make use of this support to provide much of the power that the 
   Aprobe "events.ual" probe provides to command-line users.
3. The UALs tree in the RootCause Main window has been "cleaned up", and 
   the RootCause Help menu now provides a "UALs" item that describes
   each of the probes in the UALs tree.
4. A "Find Text In Source" popup menu item has been added in the
   source pane.  Just right-click (with MB3) in the upper-right pane 
   when source is visible to see the item, and click on it to bring
   up a dialog.
5. A "Copy" action has been added to the  Event Tree and Details Tree
   in the Trace Display.  Just select nodes by clicking (with normal
   multi-select behavior by using the Control and Shift keys), then
   right-click to show the popup menu.  Click on Copy, and this will 
   copy an image of the selected nodes to the primary clipboard.  
   You can then use the "xclipboard" utility to access the copied text.
6. The NO ACTION default item in the Probes tab in Trace Setup has been
   removed, so the default is now Function Entry.
7. A problem re-opening a workspace on a smaller display has been fixed.
8. Performance of the Trace Display has been improved.
9. Shadow include directories are now included by default when parsing the
   header files so you can more easily create your own shadow headers.

10. Problems with the list of APD files being incomplete have been fixed.

Features and Fixes in Aprobe 4.3.0

1. A new probe (java_memstat.ual) is now delivered for analyzing Java object 
  leaks. This utilizes the JVMPI interface provided by the Java virtual
   machine to statistically sample and analyze java object growth and pinpoint
   problem objects.
2. The existing memstat probe has been substantially improved to provide
   many of the same features as the new Java object leak probes. It too
   can now sample and statistically analyze the memory usage of applications
   that use the standard malloc and free system calls. 
3. An interface UAL to the JVMPI is now available. Its specification is in
   libapjvmpi.h but a working knowledge of the JVMPI interface is recommended
   to make use of it. 
4. The logged data has been optimized a little better to increase the number
   of traced items that can be stored in each data file.
5. The heap.ual predefined probe that some of our customers have grown
   attached to over the years is now delivered again, albeit in an
   undocumented state.
6. New APIs for creating a vector from a hash table have been added.
7. Support for getting symbol and debug information from archive files
   has been added to 'apc' and 'apcgen'.