OCS Release Notes Aprobe 4.5.4 AIX64

From OC Systems Wiki!
Jump to: navigation, search

                          *READ THIS FIRST*

                          Release Notes for
    RootCause Version 2.3.4-64 (Aprobe Version 4.5.4-64) for AIX
                            Sep 15, 2021

This file contains brief installation and release notes for 
RootCause 2.3.4-64 and Aprobe 4.5.4-64 for AIX.  

This is a variant of the standard RootCause 2.3.4/Aprobe 4.5.4 built
especially for 64-bit applications on AIX.  Full documentation shipped
with the 32-bit version applies to this version, except where noted

Features and Fixes in RootCause 2.3.4/Aprobe 4.5.4 for 64-bit AIX applications
 (2691) add probe to log memory for hex dump at format
 (2692) create a call path profile probe
 (2695) memcheck does not record C++ allocations
 (2698) Apcentry crashes handling PA variant records
 (2699) add AP_MAKE_CPP_STRING to cppstring.h/c
 (2706) traced inlined code line numbers confusing
 (2713) remove help output from coverage output
 (2716) AIX can't set member field in class type
 (2720) xlC base class offset bug (too large)

Features in prior releases are listed below (look for "PRIOR").

Getting Help

If you have any questions or problems, please contact OC Systems by
e-mail at support@ocsystems.com, or by phone at 703-359-8160.


All OC Systems product documentation is available at 

The RootCause Console 'Help' menu points to the OCS Web site, as
well as the full documentation delivered with the product.


Product Installation and Licensing is described in Chapter 2 of the
RootCause User Guide and in the file "install.txt" located in the root
directory of the CD. 

Once installation is complete, you must source the RootCause/Aprobe setup
script to use the product. To do this, run:

  . <install_dir>/setup             (for ksh, bash)
  source <install_dir>/setup.csh    (for csh)

After you have successfully installed RootCause, you should work
through the Demonstration Program described in Chapter 5 of the
RootCause User Guide.

If you are already a user of Aprobe, you can assume that this version
works the same.


This release is source-code compatible with previous
versions, so you shouldn't need to change any APC files. However,
this version is NOT binary-compatible, with respect to UAL files.
That means you must recompile your probes from their APC source.

This release is NOT binary compatible with any 32-bit releases.

This version of RootCause/Aprobe runs only on AIX 5.3 or newer.

This version of RootCause/Aprobe only Java probes programs running with the
IBM Java 64-bit JVM versions 5, 6 or 7.

The RootCause Console (as opened by the 'rootcause open' command) requires
a 64-bit Java installation, version 1.4 or higher. This is not provided 
as part of RootCause, but may be downloaded from IBM. For example, see

Using 64-bit RootCause

If all the applications, including Java, with which you'll be using
RootCause are 64-bit object files, then this product will work just
like the 32-bit version documented in the user's guide:
and the FAQ:

Use the 'file' command to identify whether a binary is 32- or 64-bits.
For example, the following are 64-bit applications that are compatible
with this version of RootCause/Aprobe:

$ file test.exe
test.exe: 64-bit XCOFF executable or object module not stripped

$ file $(which java)
/usr/java15_64/jre/bin/java.exe: 64-bit XCOFF executable or object module not stripped

If the 'file' output does not say 64-bit then the applications are _not_
compatible with this version, and you must use the 32-bit version of
Aprobe instead.

Using Both 32- and 64-bit RootCause

This 64-bit version of RootCause supports only 64-bit applications, and
requires a 64-bit Java Runtime for the UI.  The RootCause Console can
only be used to instrument 64-bit applications.  However, 64-bit AIX
can also run 32-bit applications, and a separately-installed 32-bit version 
of RootCause may be used to instrument and analyze those.

Generally, it's simplest to use 64- and 32-bit versions from separate
command-line environments that can have one or the other in the local
PATH definition.  However, it is possible to intersperse them within
the same environment, because they use different environment variables:

For compatibility, 32-bit Aprobe continues to use APROBE, APROBE_REGISTRY, 
APROBE_LOG, APROBE_JRE, etc., while 64-bit Aprobe uses APROBE64, APROBE64_REGISTRY, etc.
The one "snag" is PATH, since the programs themselves, such as rootcause and
aprobe, are the same in both.   To work around this, we supply two "wrapper"
scripts, ap32 and ap64.   To use both in a primarily 64-bit environment,
you would do the following:

# set up 32-bit Aprobe environment first
. /opt/aprobe/setup 

# set up 64-bit Aprobe environment (goes at the front of the PATH)
. /opt/aprobe64/setup 

# open a rootcause workspace for your 64-bit application
rootcause open 
rootcause register -x my64bitapp.exe -w my64bitapp.aws
rootcause run my64bitapp.exe 

# do the same for your 32-bit app:
ap32 rootcause open 
ap32 rootcause register -x my32bitapp.exe -w my32bitapp.aws
ap32 rootcause run my32bitapp.exe 

Of course, if you are usually working with a 32-bit application on 
your 64-bit AIX, you can do the opposite:

# set up 64-bit Aprobe environment first
. /opt/aprobe64/setup 

# set up 32-bit Aprobe environment in front
. /opt/aprobe/setup 

# probe a 32-bit application as normal:
aprobe -u trace my32bitapp.exe

# probe a 64-bit app by prefixing the command with ap64:
ap64 aprobe -u trace my64bitapp.exe

Features and Fixes in RootCause 2.3.3/Aprobe 4.5.3 for 64-bit AIX applications
 (2667) AIX symbol demangling fooled by file sec.C
 (2668) Allow patching PPC trap instructions
 (2669) AIX Aprobe can't find start address in stripped exe
 (2677) AIX variant record size computed incorrectly
 (2678) Create probe to track (large) memory operations
 (2679) hmtb.ual doesn't show tracebacks

Features and Fixes in RootCause 2.3.2/Aprobe 4.5.2 for 64-bit AIX applications
 (2637) Coverage snapshot missing for forked pids
 (2639) AIX: can't probe stripped executable (Adaentry)
 (2640) statprof report improvements
 (2641) aprobe core dump on Xbarco Gcc app
 (2645) Create apcgen -U type dictionary.
 (2647) hmp report label MaxOutstanding not clear
 (2649) create HMTB - heap memory traceback probe
 (2651) prototype using a overloaded function selector

Features and Fixes in RootCause 2.3.1/Aprobe 4.5.1 for 64-bit AIX applications
 (2219) More helpful error message(s) when no reachable patch area
 (2582) Aprobe AIX does not demangle PowerAda operators
 (2604) enhancements to memcheck: -S -e -E options
 (2610) Add pragma trap_patch to allow a trap patch on PPC
 (2613) provide mechanism to register dll/so with Aprobe
 (2616) small return type does not set entire reg
 (2617) branch coverage report should show which tests take branch
 (2619) Linux/AIX ap_StubRoutine on_line/offset does not work
 (2620) memcheck should support resetting freed block values
 (2621) Add indication of inlined function to apcgen -D
 (2628) track taken test names in brcov
 (2636) statprof -a arguments parsed incorrectly on AIX64

Features and Fixes in RootCause 2.3.0/Aprobe 4.5.0 for 64-bit AIX applications
 (2552) statprof -a timings off due to bogus extra entry
 (2554) AIX boolean out parameter not set right on_exit
 (2558) COVERFILE should exclude compiler generated functions
 (2559) include date in statprof timestamp
 (2560) APO file doesn't handle single quote for options
 (2561) coverage does not do branches for some functions
 (2564) struct with ap_TimeT fields get false unions
 (2566) out of memory reading XCOFF to patch program exit actions
 (2572) add heap memory profile (hmp) probe

Features and Fixes in RootCause 2.2.9 for 64-bit AIX applications
 (2510) brcov.cfg missing from delivery
 (2511) add ability to prune ADI files 
 (2517) merge brcov into coverage and extras
 (2521) add early Aprobe debug
 (2527) default name matching for .adi files not convenient
 (2528) apbc should process similar .brc files
 (2529) memleak and memwatch should have delayed start
 (2530) run axdln on .brc file when written by apformat
 (2531) apcgen reads all debug even if -f used
 (2532) AIX sigaction instrumentation causes memory problem
 (2533) coverage.ual core dumps if -if used
 (2535) apbc doesn't handle similar .brc files with .eab
 (2539) provide option to set statprof interval
 (2540) Upgrades to apcgp.
 (2541) abrmerge should warn if source file not found
 (2544) ppc branch coverage causes core dump with -O
 (2545) add statprof option to profile address range

Features and Fixes in RootCause 2.2.8 for 64-bit AIX applications
 (2492) axdln doesn't tolerate # lines
 (2493) apc rejects -unit name with mixed case
 (2494) apc should prefer local symbol if file scope given
 (2495) Further branch coverage improvements
 (2497) apcgen -T and -M don't work
 (2498) AIX _exit() from thread doesn't trigger program on_exit
 (2505) create a name demangling tool
 (2508) AIX apcgen prints wrong decl file/line

Features and Fixes in RootCause 2.2.7 for 64-bit AIX applications
 (2449) brcov doesn't handle lines from multiple files
 (2451) support enable/disable calls and apdemand in memstat
 (2453) brcov improvements
 (2454) COVERFILE files cause core dump when inserting multiple files
 (2457) apbc.pl doesn't correctly mark some lines
 (2458) AIX Apcentry ap_StrFree coredump in CloseCoffMod 
 (2459) AIX Apcentry coredump in SetCurrentStatement
 (2461) apcgen -La reads all debug info
 (2462) apbc/bc.pl does not handle unprobed lines in .bc
 (2463) coverage does not handle fork() well
 (2467) apcgen and apc should declare maxdata:0x40000000
 (2472) typo in error message
 (2474) Aprobe setup script must check for all of Aprobe
 (2484) -qmin_heap breaks memstat traceback table

Features and Fixes in RootCause 2.2.6d for 64-bit AIX applications
 (2053) probe calling ap_FileExists causes Ada exception
 (2374) $APROBE/setup should update library path
 (2375) Warn if probing system calls
 (2376) log fixed-length string return value fails
 (2378) partial coverage percentages not obvious
 (2388) atcmerge should print Aprobe version
 (2394) support wildcard in apcgen -p argument
 (2395) support logging local/global variables in apcgen
 (2396) make ExpandDottedLines the default for coverage
 (2397) support COVERFILE 'sourcefilename' in coverage.cfg
 (2400) axdln -s doesn't work on specs
 (2401) Coverage should report reasons functions not probed
 (2404} Warn about mismatched function version in coverage
 (2405) atcmerge should allow merging diff func versions
 (2406) apbc should warn if no source files found/given
 (2409) atcmerge should produce .bc file as an option
 (2411) Allow aliases of module names (libc.so.6)
 (2415) Coverage should report maximum offset count for line count
 (2423) Add source file matching option to apcgen
 (2424) apcgen generated incorrect scope for func-local var
 (2426) Problem stubbing abort() with coverage.ual
 (2427) Need mechanism to specify function not to sacrifice
 (2430) provide access to STL exception message
 (2431) exceptions.ual should provide Ada/C++ exception message string
 (2437) on_exit probe breaks cross-lang exception raise
 (2439) ppc sacrifice function with large TOC
 (2441) memstat statistics are displayed incorrectly
 (2443) provided traceback blacklist address support
 (2444) AIX 64 core dump reading memstat cfg file

Features and Fixes in RootCause 2.2.6c for 64-bit AIX applications
 (2368) coverage.ual reports empty non-executed lines for 
 (2370) wrong symbol filename used due to #include

Features and Fixes in RootCause 2.2.6b for 64-bit AIX applications
 (2320) AIX C++ coverage included lines/files wrong

Features and Fixes in RootCause 2.2.6 for 64-bit AIX applications
1. (2268) can't set float return value in on_exit after stub
2. (2276) AIX apinfo core dump AddressOfCallToMain
3. (2303) AIX atcmerge core dump, 0 length FT (E74103)
4. (2304) axdln doesn't handle nested generics
5. (2305) AIX apcgen -D prints wrong line number for symbol
6. (2308) AIX fail to handle Ada function named operator (E74146)
7. (2309) AIX axdln must tolerate C++ symbols and old-style input
8. (2313) AIX C++ global symbol not demangled
9. (2314) apinfo -h dumps core on AIX

Features and Fixes in RootCause 2.2.5d for 64-bit AIX applications

 1. (1361) add changes, comments to coverage.2/bc.pl
 2. (1477) coverage sometimes fails to report missed lines
 3. (1516) AIX aprobe reports wrong lines for inlined C++
 4. (2203) coverage.cfg should allow inline comments
 5. (2221) AIX apc logs wrong registers for xlC/ C++ parameters
 6. (2228) implement trap patches on AIX PPC.
 7. (2240) no covearge snapshot on SIGINT or exception
 8. (2241) atcmerge fails merging .tc files from multiple apps
 9. (2242) apcgen -f doesn't handle include files
10. (2244) apcgen misses C++ members fields in big class
11. (2249) optionally reset counts between coverage snapshots
12. (2250) each coverage snapshot should write separate .tc and .bc
13. (2253) coverage summary report line should put symbol last
14. (2254) coverage CommaOutputFile gets duplicate content
15. (2255) implement ap_LoadAllSymbols for AIX...
16. (2256) improvements to coverage reports
17. (2270) report all default options in coverage cfg file

Features and Fixes in RootCause 2.2.5c for 64-bit AIX applications

 1. (2174) aprobe cores loading UAL that depends on a .so
 2. (2176) aprobe should give more info when ual fails to load
 3. (2195) ADI files from same src should be bitwise identical
 4. (2205) problem parsing gcc 5.x attribute list
 5. (2207) ap_DoCoverageSnapshot() doesn't work
 6. (2212) crash in AIX Aprobe64 signal handler (>63)
 7. (2217) apc driver should not invoke user's .kshrc file
 8. (2218) Provide a way to increase density of patches on AIX
 9. (2219) More helpful error message(s) when no reachable patch 
10. (2227) system_ual.apc cannot compute StartAddress

RootCause 2.2.5b

 1. (2139) lines not logged if duplicate timestamp
 2. (2141) apc parse error parsing complex gnu C type decl
 3. (2142) AIX apjava doesn't work on java6_64
 4. (2145) AIX crash starting IBM MB with aprobe
 5. (2146) AIX aprobe on java6_64 causes memory corruption
 6. (2152) AIX apcgen doesn't recognize .o as being C++ if -qnotimestamps
 7. (2155) optionally disable __ThrowV6 probe
 8. (2160) memwatch example broken on AIX
 9. (2161) java_memstat, java_exceptions no longer supported
10. (2164) java aborts under RC trace
11. (2170) aix64 crash unwinding native to Java

RootCause 2.2.5

 1. (2069) unhelpful message for $a[i, j]
 2. (2074) AIX RC Trace Setup doesn't show lines, modules
 3. (2114) AIX apjava doesn't recognize java7
 (All these also applied to the 32-bit version.)

RootCause 2.2.4a

The focus of this delivery was compatibility with AIX 7.  
This version also introduces a new "demand.ual" predefined
probe, and fixes some RootCause UI problems.  Specifically:

 1. (1966) apjava fails on latest IBM Java5
 2. (2023) aprobe doesn't install gcc on aix6,7
 3. (2028) apc can't parse unistd.h on aix7
 4. (2030) rootcause crashes on AIX7 expanding modules
 5. (2034) add a general user demand interface to apc
 6. (2036) memstat gui closes prematurely
 7. (2037) AIX RC 'Add Dynamic Library' should handle libX.a(shr.o)
 8. (2038) Update RC/Aprobe FAQ 
 9. (2044) AIX apc gens wrong code to deref short C++ ref param
10. (2047) AIX64 bad $struct param addressing

All but issue 2047 also apply to the 32-bit version of the product.
More generally, RootCause Version 2.2.3a-64 (Aprobe 4.4.3a-64) is the same as 
RootCause Version 2.2.3a (Aprobe 4.4.3a) with these exceptions:

1. It supports only 64-bit AIX native and Java programs.

2. It uses a different mechanism for Java instrumentation which is
   supported only in JVM 1.5 or newer.  How this changes what can
   and cannot be instrumented, or accessed from your Java probes, 
   is still under investigation.

3. The installation location is defined by the environment variable
   APROBE64, and the RootCause user data is defined by the environment
   The value of APROBE64_HOME defaults to $HOME/.rootcause_aix64
   All other product-related environment variables are unchanged. 

New Feature: demand.ual

A new predefined probe, "demand.ual", along with its header file
demand.h and supporting command-line tool apdemand, provide a
framework for "demanding" action from another probe, from the
command-line at run-time, independent of what the probed program might
be doing.  This is an advanced feature, but can be powerful in the
right circumstances.   To learn more, copy the example directory
$APROBE64/examples/predefined_probes/demand to a working area, and start
with the README file there.  It shows how you can use demand.ual in
conjunction with profile.ual to take a performance snapshot 
"on demand".  The "RemoteControl" file in the same directory
provides more detail in how to use this to control your own probes
and applications.  Contact support@ocsystems.com if you have questions.

Known Problems & Restrictions

As with any software product, we know of lots of things we'd like to
improve, but you probably won't notice or encounter them.  Some things
you might see are:

 1. Except where contradicted below, the same limitations exist in this
    version as in the 32-bit version.

 2. Only AIX version 5.3 and above is supported.   
    Contact support@ocsystems.com if you need support for an older version.

 3. ONLY 64-bit applications are supported by this version of RootCause or 

 4. Code compiled with newer versions of gcc or g++ can generally be probed.

 5. Only the IBM 1.5 or newer 64-bit JVM is supported.

 6. Java "boot classes" _can_ be traced, but other Java probes which 
    may have lost direct visibility to data loaded from other class-loaders.
    Contact OC Systems for guidance here.

If you have other problems please contact us at support@ocsystems.com.