OCS Release Notes Aprobe 4.5.4ai Linux32

                                   
                          *READ THIS FIRST*

                          Release Notes for
              RootCause Version 2.3.4ai (Aprobe Version 4.5.4ai)
                  for 32-bit Linux/x86 Applications
                             Sep 27, 2021
                                   
                                   
This file contains brief release notes for RootCause 2.3.4ai and Aprobe
4.5.4ai for 32-bit Linux x86 platforms.  Changes are relative to RootCause 
version 2.3.4 (Aprobe 4.5.4) released Sep 2021.

Features and Fixes in RootCause 2.3.4ai/Aprobe 4.5.4ai for (32-bit) Linux
-----------------------------------------------------------------------
 (2721) statprof -r option produces odd resutls
 (2722) apcgen core dumps reading DWARF variant with no disc
 (2723) Handle Linux reg param for inlined func

Features in prior releases are listed below (look for "PRIOR").


Getting Help
------------

If you have any questions or problems, please contact OC Systems by
e-mail at support@ocsystems.com, or by phone at 703-359-8160.

Documentation
-------------

All OC Systems product documentation is available at 
http://docs.ocsystems.com.

The RootCause Console 'Help' menu points to the OCS Web site, as
well as the full documentation delivered with the product.

The RootCause User Guides are available in PDF format in the files
RootCauseJava.pdf and RootCauseCpp.pdf in the product installation
directory.


Installation
------------

Product Installation and Licensing is described in Chapter 2 of each
RootCause User Guide, and in the "install.txt" file located in the
root directory of the CD.  The installation process has not changed in
this update.

After you have successfully installed RootCause, you should work
through the Demonstration Program described in Chapter 5 of the
RootCause User Guide.

Supported Versions
------------------

All support is for 32-bit applications (native or Java only), although
they can be running on 64-bit linux kernel.

For 64-bit applications, you must download and install the corresponding 
64-bit version. 

We support and have tested with Redhat versions 9.0, Red Hat Enterprise 2, 3,
4, 5, 6, 7; and corresponding Centos and Fedora Core versions.  
We fully support C and C++ programs compiled with GCC version 3.x and 4.x 
compilers using -g.  We support Java programs 
compiled with Sun Java 1.4 or newer.

This release fully supports 32-bit programs compiled with public GNAT
4.4.7 and the latest versions of GNATPro GNAT (currently 7.4.1 and
17.0 wavefronts).


Compatibility
-------------

This release is source-code compatible with previous
versions, so you shouldn't need to change any APC files. However,
this version is NOT binary-compatible, with respect to UAL files.  
That means you must recompile your probes from their APC source.

Libiberty.so
------------

The libiberty.so shared library is delivered under the terms of the Lesser GPL.
A description of this, and a copy of the license, can be found in
README.libiberty.


Libelf.so
---------

This version of RootCause requires libelf.so.1 to be in your library
path.  If you don't have this contact support@ocsystems.com and we'll
help you find one for your linux version.


Dynamically-loaded (dlopen) libraries
-------------------------------------

Tracing a "dynamic module" C/C++ library causes it to be statically
loaded at program start (via aprobe using -dll command).  This means
that one must be sure that the library specified in the RootCause
workspace must be the exact same one which the program would normally
load.  Furthermore, this won't work at all for libraries whose
initialization depends on program state. Any -dll libraries that cause
libc.so to be loaded at a different address than defined in the
program header will cause problems.

Setuid Applications
------------------- 
Applications which are owned by root and have the "setuid" bit set
require special consideration.  Contact support@ocsystems.com
for assistance.

Java
----
Java probes written completely differently than native (C/C++/Ada)
probes.  See the User's Guide and/or contact support@ocsystems.com about
probing Java.

Logging data over NFS
---------------------
Aprobe's log mechanism uses memory-mapped APD files to achieve its excellent
performance.  When the APD files are located on an NFS-mounted disk, 
corruption or loss of data logged to multiple APD ring files may occur. 
This has always been the case, and Aprobe and RootCause documentation 
encourages users to log to files on local (or at least non-NFS-mounted) disks.   
However, it can be very confusing when it does happen and you don't
realize it, so we introduced a warning (only with aprobe -v) and a
mechanism to avoid the problem.  The mechanism is enabled by exporting
the environment variable AP_NFS_TMP=1 before running aprobe.  Aprobe
will recognize when NFS logging is taking place, and instead log under
/tmp. When program execution is complete, the apd files are copied to the
final destination.  Until this copy occurs, the APD file is instead a text
file that explains what's going on.  If you would like more information,
or have problems using this feature, please contact support@ocsystems.com.


---------------------------------------------------------------------------
               FEATURES INTRODUCED IN PRIOR RELEASES
---------------------------------------------------------------------------

You may be interested in these features/changes introduced in 
prior releases of RootCause and Aprobe.

Features and Fixes in RootCause 2.3.4/Aprobe 4.5.4 for (32-bit) Linux
-----------------------------------------------------------------------
 (2688) Apc compiler fails to convert class with incomplete base type
 (2689) Handling of virtual base classes is problematic
 (2691) add probe to log memory for hex dump at format
 (2692) create a call path profile probe
 (2694) branch coverage warning about unknown opcode
 (2695) memcheck does not record C++ allocations
 (2699) add AP_MAKE_CPP_STRING to cppstring.h/c
 (2704) Linux32 does not handle fixed point type
 (2705) Gnat 22/Gcc 10 changed subprogram overload suffixes
 (2706) traced inlined code line numbers confusing
 (2707) only print Gnat [nnnn] suffix waring in verbose mode
 (2708) bad layout for Ada record with dynamic fields
 (2709) Gnat 22.0w4 runtime exception signals changed
 (2710) Gnat 22.0w4 uses variant/variant_part tags
 (2711) bad layout for repc spec'd variant type
 (2712) filter compiler generated functiosn in Gnat 22
 (2713) remove help output from coverage output
 (2715) LINUX 32 DWRF query ap_dwarf_loclist_n fails
 (2718) x87 float registers (ST(n)) not correct
 (2719) GNAT 22 exception names have @CXXABI suffix

Features and Fixes in RootCause 2.3.3/Aprobe 4.5.3 for (32-bit) Linux
-----------------------------------------------------------------------
 (2678) Create probe to track (large) memory operations
 (2679) hmtb.ual doesn't show tracebacks
 (2681) Allow more flexible call patches (on Linux).
 (2686) Aprobe fails on RHEL8 program wiht dlopen() exceptions

Features and Fixes in RootCause 2.3.2/Aprobe 4.5.2 for (32-bit) Linux
-----------------------------------------------------------------------
 (2637) Coverage snapshot missing for forked pids
 (2640) statprof report improvements
 (2642) Gnat20/Gcc8 optimized code hard to probe
 (2645) Create apcgen -U type dictionary.
 (2646) union field type not found in apc.
 (2647) hmp report label MaxOutstanding not clear
 (2648) Apcentry can't handle struct with field of stub type
 (2649) create HMTB - heap memory traceback probe
 (2650) Struct field references don't compile in apc
 (2651) prototype using a overloaded function selector
 (2654) ap_ThrowGccString does not work with -q min_heap

Features and Fixes in RootCause 2.3.1/Aprobe 4.5.1 for (32-bit) Linux
-----------------------------------------------------------------------
 (2592) COVERFILE should exclude (more) compiler generated func
 (2604) enhancements to memcheck: -S -e -E options
 (2610) Add pragma trap_patch to allow a trap patch on PPC
 (2613) provide mechanism to register dll/so with Aprobe
 (2614) Linux compiler-generated member names with special char
 (2615) core dump using ap_GnatRaiseException
 (2616) small return type does not set entire reg
 (2617) branch coverage report should show which tests take branch
 (2618) Linux use of ap_StringValue() in on_entry fails to compile
 (2619) Linux/AIX ap_StubRoutine on_line/offset does not work
 (2620) memcheck should support resetting freed block values
 (2621) Add indication of inlined function to apcgen -D
 (2622) ap_ThrowGccException does not work on Linux
 (2623) Gcc optimized hot/cold funcitons have no debug info
 (2626) apc compile error: Expecting an operand of a numeric type RHEL8
 (2628) track taken test names in brcov
 (2629) stdlib.h compile error using Apcentry on REHL8
 (2630) apc parse error on slice range with Gcc 8
 (2631) fail apc compile of __alignof__ keyword on RHEL8
 (2632) Aprobe fails at startup on RHEL8
 (2633) Gnat exceptions not working with exit probes
 (2634) ap_RaiseGnatException() not working on_line
 (2635) ap_ThrowGccException() not working on_line

Features and Fixes in RootCause 2.3.0/Aprobe 4.5.0 for (32-bit) Linux
-----------------------------------------------------------------------
 (2552) statprof -a timings off due to bogus extra entry
 (2557) statprof -c fails on linux64
 (2558) COVERFILE should exclude compiler generated functions
 (2559) include date in statprof timestamp
 (2560) APO file doesn't handle single quote for options
 (2561) coverage does not do branches for some functions
 (2562) Linux Gcc try harder to find exception name/message
 (2563) Gnat/Gcc parameters not referenced correctly when -O
 (2564) struct with ap_TimeT fields get false unions
 (2567) x86/64 disassemlber sets float mem ins class OTHER
 (2568) Linux/Gnat Apcentry doesn't allow target type expr outside probe
 (2569) Linux/Gnat dynamically-sized struct fields not generated
 (2570) apcgen core dump processing optimized Gnat code
 (2572) add heap memory profile (hmp) probe
 (2574) Handle Gnat optimized symbol names
 (2575) Gnat optimized code has formals in lexical block

Features and Fixes in RootCause 2.2.9/Aprobe 4.4.9 for (32-bit) Linux
-----------------------------------------------------------------------
 (2510) brcov.cfg missing from delivery
 (2511) add ability to prune ADI files 
 (2512) Apc generates bad layout for variant record 
 (2513) apc core dump when emiting warning message
 (2515) support GNAT Pro 19.0
 (2516) apc can't compile gcc std::tuple with formal parm __ele
 (2517) merge brcov into coverage and extras
 (2518) problem with GNAT exception propagation
 (2520) Linux gcc problem reading static member variable
 (2521) add early Aprobe debug
 (2522) fail to handle gcc const variables
 (2523) DWARF2 has no location for extern var
 (2524) failure to find g++ sepcial symbols
 (2525) x86 support for XMM registers
 (2527) default name matching for .adi files not convenient
 (2528) apbc should process similar .brc files
 (2529) memleak and memwatch should have delayed start
 (2531) apcgen reads all debug even if -f used
 (2533) coverage.ual core dumps if -if used
 (2535) apbc doesn't handle similar .brc files with .eab
 (2537) g++ const value uses 4 byte block
 (2539) provide option to set statprof interval
 (2540) Upgrades to apcgp. 
 (2541) abrmerge should warn if source file not found
 (2545) add statprof option to profile address range
 (2546) problem probing __x86.get_pc_thunk.XX() on Linux
 (2547) gcc 7.3.1 C++ symbols not demangled
 (2549) Linux params/locals not correct on_line

Features and Fixes in RootCause 2.2.8/Aprobe 4.4.8 for (32-bit) Linux
-----------------------------------------------------------------------
 (1197) Linux: exit() from thread doesn't trigger program on_exit
 (2490) provide axdln for Linux
 (2492) axdln doesn't tolerate # lines
 (2493) apc rejects -unit name with mixed case
 (2494) apc should prefer local symbol if file scope given
 (2495) Further branch coverage improvements
 (2496) GNAT Linux reference to inout param in stubbed function wrong
 (2497) apcgen -T and -M don't work
 (2499) apmkadi on Linux checksum relies on stripped item
 (2501) core dump on stripped Linux executable
 (2505) create a name demangling tool
 (2507) compile errors in APC compile for array struct members

Features and Fixes in RootCause 2.2.7/Aprobe 4.4.7 for (32-bit) Linux
-----------------------------------------------------------------------
 (2449) brcov doesn't handle lines from multiple files
 (2450) cannot log GNAT static-sized record subtype
 (2451) support enable/disable calls and apdemand in memstat
 (2453) brcov improvements
 (2454) COVERFILE files cause core dump when inserting multiple files
 (2457) apbc.pl doesn't correctly mark some lines
 (2460) apcgen warnings fo C++ symbol _Z
 (2461) apcgen -La reads all debug info
 (2462) apbc/bc.pl does not handle unprobed lines in .bc
 (2463) coverage does not handle fork() well
 (2465) Trap patch on line at end of function not made
 (2469) apc problems with wavefront 18.0w 20170628
 (2471) GNAT 18.0 w20170601: main-nested subp debug info not fo
 (2472) typo in error message
 (2474) Aprobe setup script must check for all of Aprobe
 (2476) Upgrade i386 disassembly
 (2477) apc generates incorrect layout for repspec'd variant record
 (2480) no line information in Gnat shared library
 (2481) Linux libdal.so does not support preinit
 (2484) -qmin_heap breaks memstat traceback table
 (2485) apc defines unknown types for type nested in subp decl

Features and Fixes in RootCause 2.2.6d1/Aprobe 4.4.6d1 for (32-bit) Linux
-----------------------------------------------------------------------
 (2464) GNAT exception reraise changed for wavefront 18.0w 2017
 (2468) aprobe setup script doesn't handle blanks in PATH

Features and Fixes in RootCause 2.2.6d/Aprobe 4.4.6d for (32-bit) Linux
-----------------------------------------------------------------------
 (2373) wrong filename scope for GNAT symbols
 (2374) $APROBE/setup should update library path
 (2375) Warn if probing system calls
 (2376) log fixed-length string return value fails
 (2378) partial coverage percentages not obvious
 (2379) Linux target expression problems
 (2382) Linux/Gnat exception reason string not right
 (2383) Add support to raise C++ exception from probe
 (2384) Linux/Gnat position of fixed-length string params wrong
 (2387) emit DWARF warning messages in verbose mode
 (2388) atcmerge should print Aprobe version
 (2390) support gnatpro 18.0w for APTs
 (2393) support apcgen -D for big Linux executables
 (2394) support wildcard in apcgen -p argument
 (2395) support logging local/global variables in apcgen
 (2396) make ExpandDottedLines the default for coverage
 (2397) support COVERFILE 'sourcefilename' in coverage.cfg
 (2399) fix logging Linux return structs
 (2401) Coverage should report reasons functions not probed
 (2402) Gnat 18.0w DWARF subtypes before base type
 (2404} Warn about mismatched function version in coverage
 (2405) atcmerge should allow merging diff func versions
 (2406) apbc should warn if no source files found/given
 (2407) Warn if Gnat [nnn] suffix not needed
 (2409) atcmerge should produce .bc file as an option
 (2411) Allow aliases of module names (libc.so.6)
 (2413) Aprobe Linux32 stat/fstat/lstat on 64-bit file system
 (2415) Coverage should report maximum offset count for line count
 (2423) Add source file matching option to apcgen
 (2424) apcgen generated incorrect scope for func-local var
 (2425) apc can't reference Gnat out params when optimized
 (2426) Problem stubbing abort() with coverage.ual
 (2430) provide access to STL exception message
 (2431) exceptions.ual should provide Ada/C++ exception message string
 (2432) move DWARF warnings to verbose mode
 (2435) Linux/Gnat tag field is hidden when logging
 (2436) Linux on_exit probe breaks cross-lang exception raise.
 (2441) memstat statistics are displayed incorrectly
 (2443) provide traceback blacklist address support
 (2445) Linux/Gcc handle cxx11 tag when demangling names
 (2455) apc generated bad code for Gnat in array param ref

Features and Fixes in RootCause 2.2.6c/Aprobe 4.4.6c for (32-bit) Linux
-----------------------------------------------------------------------
 (2370) wrong symbol filename used due to #include
 (2371) Linux aprobe fails to demangle 'gnat170w9' symbols

Features and Fixes in RootCause 2.2.6b/Aprobe 4.4.6b for (32-bit) Linux
-----------------------------------------------------------------------
 (2280) Linux Aprobe problem with LD_PRELOAD in forked process
 (2300) Linux/GNAT upscope references not supported
 (2318) bc support for linux
 (2319) Linux/Gnat exception reason string not reported.
 (2321) sigsegv.ual causes core dump on Linux386
 (2322) Linux/Gnat Interfaces.C.Long_Float not handled
 (2323) Linux/Gnat records with dynamic sized components layout
 (2326) Linux apc fails when probes in same file
 (2332) Continuation of i2329 Gnat tracebacks
 (2333) Linux (F) 'thread stacks were overlapped'
 (2334) coverage.ual should do final snapshot on abort
 (2335) Linux coverage gets bad line info, causes sigsegv
 (2337) C++ const int declarations not handled
 (2338) ap_RaiseGnatException broken for Gnat shared library
 (2340) implement ap_UnprobeThread to avoid GNAT stackwalk core
 (2341) C++ exceptions not translating correctly
 (2342) GNAT abort task causes core dump unwinding
 (2346) Switch to pthread TLS support for thread tables
 (2348) Linux/GNAT: on_line probe gets wrong param value in nes
 (2349) use lower case exception name for GnatRaiseExceptionByN
 (2350) support ap_GnatRaiseException in on_entry and on_line

Features and Fixes in RootCause 2.2.6a/Aprobe 4.4.6a for (32-bit) Linux
-----------------------------------------------------------------------
 (2324) Aprobe traceback skips a frame when exit probe present
 (2329) Linux/Gnat cannot instrument lines with extended float instructions

Features and Fixes in RootCause 2.2.6/Aprobe 4.4.6 for (32-bit) Linux
-----------------------------------------------------------------------
 (2090) Linux/GNAT: update out parameter on_exit wrong
 (2243) Linux apcgen, apc should work with g++/gnat .o files
 (2259) GNAT/Linux core dump in __gnat_backtrace
 (2267) bad reference from probe for GNAT out parameter
 (2272) linux problems with ap_CallerLocation
 (2277) Linux atcmerge core dump
 (2278) Linux/GNAT: problems logging ada.bounded_string component
 (2279) GNAT/Linux apcgen wrong for param after string param
 (2282) Gnat/Linux record layout problem
 (2283) deliver libdal.so.stub for Linux
 (2285) Gnat/Linux support for backtrace(3)
 (2286) Linux/Gnat apcgen assert on char16_t predef type
 (2287) Linux/g++ DW_DLE_ATTR_FORM_BAD on huge struct
 (2288) Linux/g++ exception class ctor debug info not found
 (2289) Linux/Gnat Ada out param with cast wrong
 (2290) apc: object location cannot be determined
 (2293) Gnat/Linux access to in param on_exit
 (2295) Gnat/Linux array slice lower bound not static
 (2297) apc c constant array treated as Ada
 (2298) gnat generates REP component for partial rep spec
 (2299) Linux apcgen prints wrong source file name
 (2307) Linux/Gnat filenames wrong for template methods
 (2310) use MAP_ANONYMOUS on linux
 (2311) Linux/Gnat problems with exit probes, exceptions, Ada-C++
 (2317) deliver call graph probe apcgp.ual
 (2318) bc support for Linux

RootCause 2.2.5d/Aprobe 4.4.5d for (32-bit) Linux
-----------------------------------------------------------------------
 1. (1361) add changes, comments to coverage.2/bc.pl
 2. (1477) coverage sometimes fails to report missed lines
 3. (2203) coverage.cfg should allow inline comments
 4. (2239) Linux/386 bad messages for unkown ins type
 5. (2240) no coverage snapshot on SIGINT or exception
 6. (2241) atcmerge fails merging .tc files from multiple apps
 7. (2242) apcgen -f doesn't handle include files
 8. (2244) apcgen misses C++ members fields in big class
 9. (2246) failure for slice of array'access
10. (2248) core dump on ap_CallerLocation
11. (2249) optionally reset counts between coverage snapshots
12. (2250) each coverage snapshot should write separate .tc and .bc
13. (2253) coverage summary report line should put symbol last
14. (2254) coverage CommaOutputFile gets duplicate content
15. (2256) improvements to coverage reports
16. (2261) add experimental expand-dotted-lines tool
17. (2262) support .bc files on Linux
18. (2263) no APC type created for GNAT rep spec'd field
19. (2264) handle private incomplete types in GNAT
20. (2269) enhance statprof to profile enitre address space
21. (2270) report all default options in coverage cfg file

RootCause 2.2.5c/Aprobe 4.4.5c
------------------------------
 1. (2186) GNAT array base address computation incorrect
 2. (2189) GNAT exceptions not intercepted
 3. (2198) GNATPro/Linux: core dumps reading DWARF
 4. (2199) core dumps instrumenting Firefox
 5. (2205) problem parsing gcc 5.x attribute list
 6. (2207) ap_DoCoverageSnapshot() doesn't work
 7. (2213) Linux/GNAT CompilerKind wrong if package 'sec'
 8. (2215) Linux/Aprobe/GNAT core dump reading line information
 9. (2224) Linux assert failure in ap_CallerLocation
10. (2237) apcgen doesn't recognize C++ built w/GNAT

Features and Fixes in RootCause 2.2.5b/Aprobe 4.4.5b for (32-bit) Linux
-----------------------------------------------------------------------
 1. (2138) problems building on centos7
 2. (2139) lines not logged if duplicate timestamp
 3. (2141) apc parse error parsing complex gnu C type decl
 4. (2144) gcc4.8 requires dwarf4 support
 5. (2160) fix memwatch, memstat examples, output
 6. (2164) java aborts under RC trace

Features and Fixes in RootCause 2.2.5a/Aprobe 4.4.5a for (32-bit) Linux
-----------------------------------------------------------------------
 1. (2117) show instantiation line in apc/apcgen comments

Features and Fixes in RootCause 2.2.5/Aprobe 4.4.5 for (32-bit) Linux
---------------------------------------------------------------------

 1. (1182) on_line(last) doesn't work for generic instance
 2. (1921) avoid NFS apd logging
 3. (2051) Linux: coverage breaks I/O app
 4. (2069) unhelpful message for $a[i, j]
 5. (2085) support public gnat on linux

Features and Fixes in RootCause 2.2.4/Aprobe 4.4.4 for (32-bit) Linux
---------------------------------------------------------------------
 1. (1894) rhel6: sometimes can't instrument on_line(first)
 2. (1996) problems with trace.ual in PA/ada2cpp/g++ Linux applications
 3. (2002) PA eqOp symbol demangled wrong by RC223a


Features and Fixes in RootCause 2.2.3a for (32-bit) Linux
---------------------------------------------------------

 1. (1962) (OCS) don't retry failed demangling
 2. (1964) (OCS) provide libaprobe.adi

Features and Fixes in RootCause 2.2.3 for (32-bit) Linux
---------------------------------------------------------

 1. (1865) No module found for pa4_get_reraise_ptr(0)
 2. (1880) apcgen: '(E) ap_dwarf_loclist_n: DW_DLE_ATTR_FORM_BAD'
 3. (1885) PROGRAM_ERROR from apc refing C file in Linux PA exe.
 4. (1892) local data addressed wrong on rhel6/gcc4
 5. (1842) Linux mismatched ADI files not flagged
 6. (1884) function inside probe thread hangs
 7. (1896) apc -v shouldn't do gcc -v
 8. (1897) rhel6, handle non-standard frame base reg
 9. (1899) rhel6, Apcentry can't demangle g++ names
10. (1900) gcc4.4, g++ file-static debug info not found
11. (1911) typeof doesn't find C types in Ada binary
12. (1912) redefinition of typedef .. PtrTo_void
13. (1930) provide C++ string helper UAL

All of the above fixes also apply to the 64-bit version of the product.

Fixes 1..4 were delivered as patch 4.4.2i-p1 on November 16, 2011.

Issue 1930 introduces "cppstring.ual" which provides ap_GetCppString() and
ap_SetCppString() functions for use in your probe.  Its usage is 
illustrated in $APROBE/examples/predefined_probes/cppstring/.

Features and Fixes in RootCause 2.2.2i
--------------------------------------

 1. (1020) sigsegv stack trace incomplete
 2. (1802) port sigsegv.apc to 64-bit Linux
 3. (1804) support universal_traceback in PowerAda/Linux

Features and Fixes in RootCause 2.2.2h
--------------------------------------

 1. (1724) ap_DoPeriodically() should be cancellable
 2. (1770) support gcc4 optimized code
 3. (1773) handle some linux386 struct return logic (tzsrgsc1)


Features and Fixes in RootCause 2.2.2g
--------------------------------------

 1. (1730) fix Linux Aprobe/Java failure for JRE 1.6.0_18
 2. (1738) fix apc parse error for c99 'restrict' keyword
 3. (1756) remove ksh requirement 


Features and Fixes in RootCause 2.2.2e
--------------------------------------

 1. (1207) apformat should remember ADI files (see NOTE 1)
 2. (1386) (OCS) fix string leak in CopyNode
 3. (1434) (OCS) provide workaround for bad type layout (see NOTE 2)
 4. (1436) (OCS) refactor type replacement implementation
 5. (1441) (OCS) implement 'formatof' to manipulate apc print routines (NOTE 2)
 6. (1442) different apc results for upper, lower case
 7. (1485) formatof + typeof don't work together (NOTE 2)
 8. (1497) RC/Linux: g++ 3.4 trace setup problems
 9. (1551) Provide memleak ual for Aprobe 
10. (1561) RCLinux: aprobe DOA on FC6
11. (1570) Linux: make scripts more linux-friendly
12. (1582) JVMTI: aprobe.jar has classes not in com.ocsystems
13. (1600) Linux: aprobe runtime can't find lines in .so file
14. (1602) Update linux examples
15. (1621) ap_DoPeriodically incorrectly uses signal on threaded app
16. (1624) Linux: FC9 parse error on __signed__
17. (1625) Linux: gcc4 line numbers all wrong
18. (1626) Linux: gcc4 exceptions not supported
19. (1631) Linux: probes on libc from .h files don't work
20. (1632) Linux: profile.ual's benchmark broken
21. (1633) Linux: RC trace setup has no debug info due to libdwarf
22. (1635) Linux: do 'set +u' in setup scripts to avoid breakage
23. (1636) Linux: problems with log_env probe

  NOTE 1: apformat now "remembers" ADI files (i1207)
  
  By customer request, aprobe logs the paths of ADI files that were
  loaded at runtime, and apformat attempts to load these again at
  format-time.  apformat will not fail if they are absent, and apformat
  can be forced not to load them by renaming the .adi file itself, or
  specifying -z on the apformat command line and explicitly naming UALs
  that are wanted.  Also, apformat -u info -p "-u" ... now lists ADI
  files applied at runtime.
  
  
  NOTE 2: APC auto-formatting extensions added (i1434, i1436, i1441, i1485)
  
   This version introduces two extensions to the APC language to allow
   user-specified replacement of default apc behavior.
   
   ** typeof Declaration **
   
   The first extension is a new use of the keyword 'typeof' to designate
   a locally defined APC type to be used in place of the type that the
   apc compiler would generate to overlay the in-memory image of a
   target program's type.  For example:
   
     typedef struct
     {
        int word0, word1;
     } my_time_type;
     
     typeof $time_type  my_time_type;
   
   Then wherever an object or field of type $time_type was logged, it
   would be interpreted as type my_time_type.
   
   Why would you do this?  Mostly to work around bugs or limitations in
   the APC compiler.  The above is a real-world case where time_type was
   defined as a 'double' but the 8-byte alignment imposed by the apc
   backend compiler didn't match the 4-byte alignment used by the target
   compiler.  By replacing the 'double' with a struct consisting of 2
   ints, the alignment restriction was removed and everything worked.
   
   The onus is entirely on the user to define a compatible type; there
   is no checking for compatibility.  The original type layout must be
   determined by looking at the C code generated by the 'apc' compiler:
   
     apc -g t.apc -x t.exe  # creates t.apc_c.c file
   
   This can be further cleaned up and made readable, for example:
  
     cat t.apc_c.c | \
       grep -v "^#line" | \
       sed "s/^ *//" | grep -v "^$" | \
       indent -st -sob -dj -i3 -bl | expand -3
   
   It can still be complicated, so generally the replacement type will
   be determined in cooperation with OC Systems support.  This can then
   be defined in an include file that users can include into their
   probes as needed.
   
   
   ** formatof declaration **
   
   The typeof extension described above led to the need for a
   corresponding formatting extension, so that the replacement type
   could be printed as if it were the original, or in any other way the
   user desires:
   
      formatof type_name print_proc_name;
   
   This declaration says: when generating the automatic formatting code,
   call the already-defined function 'print_proc_name' to print any 
   object or field of type 'type_name'.   A "print proc" always has the
   prototype:
   
      void print_proc_name(int Indent, ap_NameT Prompt, type_name *Arg);
   
   For example, see the functions ap_Print_* in aprobe.h.  So, to use
   the formatof declaration to print "my_time_type" as a double value,
   you would do something like:
   
      static void 
      Print_my_time_type(int Indent, ap_NameT Prompt, my_time_type *Arg)
      { 
         double d;
         memcpy(&d, Arg, sizeof d);
         ap_Print_double(Indent, Prompt, d);
      } 
   
      formatof my_time_type Print_my_time_type;
   
   As with typeof, there is another form of 'formatof' which gives
   the existing print procedure name for a given type or expression:
   
      formatof( type_name )
   
   So you could replace the call to ap_Print_double above with:
   
      formatof(double) (Indent, Prompt, d);
   
   Yes, it looks strange, but it works.
   
   As with 'typeof', it's up to the user to ensure that the print
   routine is declared with the right prototype and does something
   sensible.  As with format routines the Arg parameter is passed by
   reference as in the example above.
   
   The major limitation we know of right now is that the 'formatof'
   mechanism doesn't work for array types, even fixed-sized ones.  
   This is because of the special handling of such types to capture the 
   size of an array as a "slice".  
   
   Aprobe historians may know that 'formatof' was originally defined
   as part of the language, for actually defining format routines.
   While the syntax was recognized, it was never implemented because it
   didn't fit the general usage of automatic formatting.  This seemed
   like a good way to capture the original intent.

Features in RootCause 2.2.2
---------------------------

 1. RootCause now supports 32-bit apps on 64-bit linux. (i735)
    64-bit applications are still not supported.

 2. Parameters of type 'long double' are now supported. (i787)

 3. Numerous improvements to the handling of ADI files. (i829, i844, i963, 
    i1188)

 4. Reduced runtime memory usage, including an option -qmin_heap which
    discards symbol table after instrumentation. (i879, i885, others)
    (See NOTE 1 below.)

 5. 'Find Function in Trace Events' now works from Event Summary Tree. (i902)

 6. The mechanism for disabling/enabling probes has been overhauled
    with more reliable mechanisms (i948). (See NOTE 2 below.)

 7. RootCause 'Build' operation now invokes prebuild.ksh, postbuild.ksh in 
    workspace if present, writing output to Message window. (i1014)

 8. Extended functionality of APROBE_DUMP_CORE_IF_ERROR. (i1018)
    (See NOTE 3 below.)

 9. Fixed crash on thread data initialization. (i1049)

10. Fixed Java 1.5 class verification error. (i1155)

11. Added 'apformat -f', analogous to 'tail -f', to format apd file(s) as they
    are written.  (See NOTE 4 below.)

12. Fixed problem with JVM and java -server invocation. (i1191)

13. Add support of IBM JVM on Linux i386. (i1204)

14. Make all -v, -!, ap_Debug() and ap_DumpMemory() output go to stderr. 
    (i1209)

15. Fixed problems with Linux runtime module location, size on RHEL4.
    (i1211, i1280)

16. Make ap_DoPeriodically() use threads when available on Linux. (i1231)

17. Removes spurious warnings when using gcc 4.x. 

18. Avoid optimization problems with params on_exit by using saved on_entry 
    params unless overridden with 'apc -y'. (i1289) 

19. Use libelf.so rather than libelf.a. (i1298)

20. Fixed a race condition in ap_HashTableDelete. (i1303)

21. Re-enabled support for DisplayReport SubtotalReport in events.apc. (i1310)




Features in RootCause 2.1.4b
----------------------------

 1. RedHat Fedora Core 3 and Enterprise Linux 4 (kernel 2.6.x) are now 
    supported.
  
 2. The function ap_GetHostId() was changed to return the same value
    as the "hostid" command instead of the license manager's host ID, 
    which is now provided by the new function ap_HostLicenseId().  
    The function ap_GetHostName() which returns the same string as 
    "hostname". (i647)

 3. The memwatch, memstat java_memstat probes now support "REMOVE"
    filters which identify allocations that should not be reported.  
    This are documented in the sample configuration files for these
    probes provided in $APROBE/probes. (i650)

 4. The ADI (Aprobe Debug Information) support has been improved in
    several ways: aprobe will look for the default-named adi file in
    the same directory as the application and will use it if it's
    checksum matches; apmkadi -v reports more information; aprobe -v
    will report whether ADI file was used and if not, why not (e.g.,
    checksum mismatch).  (i680)



Features in RootCause 2.1.4a
----------------------------
 1. The management of APD rings in the presence of frequent process creation
    was made more robust (i409).

 2. apinfo -l now prints the source file name associated with source
    lines (i538).

 3. Several improvements were made to the 'statprof' probe (i579, i585).

 4. PowerAda shared libraries are now supported by the RootCause 
    Trace Setup dialog (i581).



Features in RootCause 2.1.4
----------------------------

 1. Programs compiled with GCC version 3.x compilers with the "-g" flag
    (which generates DWARF debug information) are now fully supported.

 2. Several critical defects in RootCause deploy functionality were 
    corrected.

 3. The statprof.apc probe was ported to linux.  This probe enables
    the kernel services for performing statistical profiling on a
    given address space range.

 4. The events probe was enhanced to export JNI calls for all
    functions defined in events.h.

 5. Improvements were made in the way dot-indicators are propagated
    in the GUI's Trace Setup window.

 6. A problem where Aprobe disturbs the global "errno" value was been
    corrected.

 7. A new option (-r) has been implemented for the statprof custom
    probe that resets the counts to zero for each snapshot. This will
    result in profiling results for each interval between snapshots
    which are independent of each other, rather that the default which
    is cumulative.

 8. Java 1.5 is now supported.

 9. Several corrections were made to the custom memcheck probe.

10. A custom probe is provided to allow instrumentation of the
    de-facto standard for Java logging, log4j. This probe provides the
    ability to watch the configuration file to look for changes.

11. Handling of very large Java methods has been improved.

12. A new function is added to aprobe.h:

       extern void ap_HeapStats(ap_Uint64   *TotalBytesAllocated,
                                ap_Uint64   *TotalBytesFreed);

    This may be called any time in a probe to determine the amount of
    memory allocated by Aprobe itself, separate from the application
    being probed.  When running aprobe with the -v flag, these
    statistics are printed at various points during program startup, for
    example:

       Aprobe: Initialized thread stack
       Aprobe: Heap memory stats
               Allocated bytes: 26016
               Freed bytes: 0

13. The Aprobe run-time system and probes, as well as any user probes
    that use ap_Malloc(), now allocate heap memory from shared memory
    allocated using mmap() rather than from the program's memory space
    using malloc().  This allows Aprobe to handle applications with huge
    symbol tables and memory needs.

    The aprobe configuration parameter "-qmmap=<address>" has been
    extended to apply to all uses of mmap including this heap memory
    allocation.  Hence the specification, "-qmmap=0x20000000" would
    indicate that the first mmap operation would begin at that address
    and each subsequent mmap operation would request the next
    available page after that, whether used for APD files, aprobe
    heap, or another purpose.

    In order to allow all aprobe shared memory usage to fit into a
    single segment by default, the default size of a single APD file was
    reduced from 256M to 64M.  As always, this can be changed with the
    -s option and multiple APD files can be specified with the -s
    option.  The default size of the persistent APD file (the one
    without a numeric suffix) is reduced from 256M to 128M.  As always
    this can be changed with the -sp option.

14. When the 'apc' command fails due to a non-fatal error, and the
    option -g or -C was specified, the intermediate C file (with suffix
    .apc_c.c) is now preserved as a potential debugging aid.

15. A number of formatting improvements were made in the memstat and 
    java_memstat report files.

Features in RootCause 2.1.3a
----------------------------

 1. The Trace Events summary tree may now be pruned based upon
    statistic usage ratios between parent and child nodes.

 2. Instrumentation overhead can now be specified and reflected in
    the Trace Events summary tree.

 3. Statistics are now reported for individual nodes of the Trace
    Events summary tree.

 4. Text save/copy operations have been implemented for individual
    nodes of the Trace Events tree.

 5. Function/Method TRIGGERS have been added to the RootCause GUI.
    This mechanism allows tracing to be specified globally and
    controlled dynamically at runtime based upon function/method
    entry.

 6. Support has been added to allow probes to modify the parameters
    passed to java methods.

 7. stub support has been implemented for java methods.

 8. java run and format -time performance improvements have been
    implemented.

 9. Functions/Methods reported in the Load Shed table are no
    longer reported in the Performance Summary tables.

10. Some trace UAL options have been deprecated.

11. Various improvements have been made to tables in Trace Events.

12. Improvements have been made to the manner in which time is
    computed for recursive function calls.

13. RootCause now ships with version 1.4.2 of the java JRE.

Features in RootCause 2.1.3
---------------------------

 1. The "rootcause merge" command has been enhanced and a 
    "rootcause copy" command added to better support the use of a general
    Workspace against multiple applications.

 2. The LogTraceback method in TraceBean is now exported to facilitate
    logging throwable exceptions.

 3. The suffix ".xml" is now recognized as an alternative to ".xmj" 
    as a valid suffix for a Java Probe deployment descriptor file.

 4. Support has been added to allow dynamic Workspace selection
    based upon the instrumented application's command line arguments.
    See http://www.ocsystems.com/faqs/rc_aprobe_faq.html#q9.12.

 5. The default limit on number of trace events has been increased to
    1,000,000.  Note that since this is a per-user preference, this
    will only apply to new users or if you change your APROBE_HOME 
    directory.

 6. Trace logging performance improvements have been implemented for 
    recording machine time.

 7. .zip files discovered in the classpath are now treated just like
    .jar files.

 8. Two new pragmas are available to categorize probe targets. "pragma
    required" (the default if the module is found) identifies a probe
    target that must be present, while "pragma optional" (the default
    if the module is not found) identifies a probe target that need
    not be present. Use of these new pragmas make probes more
    readable, avoid runtime warning messages, and support the ability
    to use parameters from header files.

 9. Custom probes no longer need to explicitly check
    ap_RootCauseTraceIsEnabled to determine if tracing is enabled,
    this is now performed automatically in ap_LogSubprogramEntry/Exit.

10. Performance enhancements have been made to java method entry/exit logging.

11. memcheck.apc is now distributed. It is a memory corruption probe
    that uses a "fence" technique to detect buffer under/over runs, 
    as well as multiple deallocates.

12. Support for upcoming PowerAda 5.3.1 was added.


Features and Fixes in RootCause 2.1.2
-------------------------------------

 1. The -jar option is now supported for Java. In order for this to work, 
    the .jar file must be used to initialize the workspace (normally
    this will be automatic).  The previous workaround, changing this to
    specify the .jar file on the classpath and explicitly providing the
    main class, may be removed.

 2. Exceptions raised by C++ code compiled with gcc 3.x are now supported.

 3. Numerous fixes have been made to the handling of PowerAda debug
    information. This version only supports the new PowerAda 5.3 and
    newer.

 4. The time span displayed in the event tree is now clearly displayed.

 5. All event nodes can now be displayed in a text window for better
    integration with standard probes. This also fixes problems with
    the truncation of some useful data such as tracebacks.

 6. New statistical tables for performance analysis have been introduced,
    in place of the CALL_COUNTS tables. These provide summaries of both
    call counts and wall-clock times for methods and functions, and
    show additional statistics if provided by predefined UALs. This
    change allows out of the box profiling-style information to be viewed.

 7. A new option to log parameters for multiple Java methods has been
    introduced in the trace setup window.

 8. The deploy option is no longer licensed separately and customers who have
    any RootCause console license will be able to deploy.

 9. RootCause now correctly handles java (1.4) command lines where the JVM
    kind (server, client, etc.) is not the first parameter specified.

10. Wildcard performance in the trace setup has been improved and a number of
    fixes made to the propagation of the trace / probe markers.

11. Lines are no longer truncated in the source pane.

12. Memstat and Java memstat output in the event viewer has been greatly
    improved.

13. A liberty.so library is now delivered allowing demangling of gcc 3.x 
    C++ methods.

14. A new cached hashtable facility is now available in the runtime. See the
    ap_Cache... routines in aprobe.h for details.

15. The Memstat probes now correctly analyze the data when long runs are used
    in combination with a format start time option.

16. The Java memstat probe will disable the -parallel GC option which is not
    supported by the JVMPI interface.

17. The perf_cpu and perf_rusage probes are now included in the agent
    installation.

18. Problems related to enabling and disabling of trace have been addressed.

19. Formatting from the RootCause GUI has been fixed on RedHat 9. There appears
    to be a bug in the system call but this has been worked around.

20. A problem with the thread-stack handling code on Linux has been fixed.

21. Problems with attempting to demangle Fortran symbols as if they were Ada
    symbols have been fixed.

22. The licensing software has changed and now uses the same version across
    all platforms. The upshot of this for Linux is that new license keys are
    required.

Features and Fixes in RootCause 2.1.1
-------------------------------------

 1. You can now print graphs produced by the memstat and java_memstat
    probes.

 2. A workaround for a JVMPI bug causing OutOfMemory errors was added 
    to java_memstat.

 3. Empty jar files and empty methods are now tolerated by RootCause.

 4. You can now directly add a classpath entry via "Find Class..." 
    in the Trace Display.

 5. An incorrect ordering in the "View By Time" view in the Trace Display 
    was fixed.

 6. A problem with incorrect nesting of events in the Trace Display was fixed.

 7. The "Remove All Probes" operation in Trace Setup is now much faster.

 8. Installing RootCause from CD no longer requires 'root' privilege 
    (after the CD is mounted).

 9. Very long aprobe command-lines (e.g., due to long classpath)
    are now accepted.

10. An off-by-one problem with line numbers in tracebacks was fixed.

11. Several problems mapping PowerAda symbols to the corresponding debug
    information in the PowerAda library have been fixed.

12. RootCause now operates by default in a "Developer" mode where 
    the "Deploy" and "Decollect" operations are not supported.  An 
    additional license key is required to enable these operations.


Features and Fixes in RootCause 2.1.0
-------------------------------------

 1. The Memory Analysis Probes "memstat" and "java_memstat" have been 
    integrated with RootCause.  These probes use statistical sampling
    and analysis to collect memory and Java object usage data about your
    program with low enough overhead to use in real-world testing.
    See http://www.ocsystems.com/user_guide/probes/memstat.html for a 
    complete description.

 2. New performance analysis support has been added to RootCause, with
    which resource data is collected at runtime and then summarized by
    thread and call-site under a new "Event Summary" node. 
    The existing trace probe, and new "perf_cpu" and "perf_rusage" probes 
    make use of this support to provide much of the power that the 
    Aprobe "events.ual" probe provides to command-line users.

 3. The UALs tree in the RootCause Main window has been "cleaned up", and 
    the RootCause Help menu now provides a "UALs" item that describes
    each of the probes in the UALs tree.

 4. A "Find Text In Source" popup menu item has been added in the
    source pane.  Just right-click (with MB3) in the upper-right pane 
    when source is visible to see the item, and click on it to bring
    up a dialog.

 5. A "Copy" action has been added to the  Event Tree and Details Tree
    in the Trace Display.  Just select nodes by clicking (with normal
    multi-select behavior by using the Control and Shift keys), then
    right-click to show the popup menu.  Click on Copy, and this will 
    copy an image of the selected nodes to the primary clipboard.  
    You can then use the "xclipboard" utility to access the copied text.

 6. The NO ACTION default item in the Probes tab in Trace Setup has been
    removed, so the default is now Function Entry.

 7. A problem re-opening a workspace on a smaller display has been fixed.

 8. Performance of the Trace Display has been improved.

 9. Shadow include directories are now included by default when parsing the
    header files so you can more easily create your own shadow headers.

10. Problems with the list of APD files being incomplete have been fixed.


Features and Fixes in Aprobe 4.3.0
----------------------------------

 1. A new probe (java_memstat.ual) is now delivered for analyzing Java object 
   leaks. This utilizes the JVMPI interface provided by the Java virtual
    machine to statistically sample and analyze java object growth and pinpoint
    problem objects.

 2. The existing memstat probe has been substantially improved to provide
    many of the same features as the new Java object leak probes. It too
    can now sample and statistically analyze the memory usage of applications
    that use the standard malloc and free system calls. 

 3. An interface UAL to the JVMPI is now available. Its specification is in
    libapjvmpi.h but a working knowledge of the JVMPI interface is recommended
    to make use of it. 

 4. The logged data has been optimized a little better to increase the number
    of traced items that can be stored in each data file.

 5. The heap.ual predefined probe that some of our customers have grown
    attached to over the years is now delivered again, albeit in an
    undocumented state.

 6. New APIs for creating a vector from a hash table have been added.

 7. Support for getting symbol and debug information from archive files
    has been added to 'apc' and 'apcgen'.