OCS Release Notes Aprobe 4.5.4c Linux64

From OC Systems Wiki!
Jump to: navigation, search
                          *READ THIS FIRST*

                          Release Notes for
               RootCause Version 2.3.4c (Aprobe Version 4.5.4c) 
                     for 64-bit Linux Applications
			     Sep  9, 2022
This file contains brief release notes for RootCause 2.3.4c (Aprobe 4.5.4c)
for 64-bit applications running on x86_64-compatible hardware.  Changes
are relative to version 2.3.4b (Aprobe 4.5.4b) dated Feb, 2022.

Features and Fixes in RootCause 2.3.4c/Aprobe 4.5.4c for (64-bit) Linux
 (2740) update libocsdwarf for DWARF5 support

Features in prior releases are listed below (look for "PRIOR").

Getting Help

If you have any questions or problems, please contact OC Systems by
e-mail at support@ocsystems.com, or by phone at 703-359-8160.


This README is currently the only documentation for specific to 64-bit
RootCause.  Most features and behaviors are as for the 32-bit
version.  All OC Systems product documentation is available at 

The RootCause Console 'Help' menu points to the OCS Web site, as
well as the full documentation delivered with the product.

The RootCause User Guides are available in PDF format in the files
RootCauseJava.pdf and RootCauseCpp.pdf in the product installation

The RootCause and Aprobe documentation refers to the APROBE environment 
variable.  When working with a 64-bit installation of Aprobe and RootCause, 
replace all uses of the APROBE environment variable with APROBE64.  The
names of the tools themselves (e.g., rootcause, aprobe) are the same in
both 32- and 64-bit installations.


Product Installation and Licensing is described in Chapter 2 of each
RootCause User Guide, and in the "install.txt" file located in the
root directory of the CD.  The installation process is the same as for
the 32-bit version, except that there is no prompt for a license key.

After you have successfully installed RootCause, you should work
through the Demonstration Program described in Chapter 5 of the
RootCause User Guide.

Supported Versions

We have tested with Redhat Enterprise and/or Centos 4, 5, 6, 7.

We fully support programs compiled with gcc and g++ version 3.x, and 4.x 
compilers.  Ada and other languages are not explicitly supported except
where their generated code and debug information is common with that 
for C and C++.


This release is source-code compatible with previous
versions, so you shouldn't need to change any APC files. However,
this version is NOT binary-compatible, with respect to UAL files.  
That means you must recompile your probes from their APC source.

NOTE: if you have a 32-bit application running on 64-bit linux, you
must use the 32-bit installation of RootCause to build the probes 
and run your application.  See "Using 64-bit RootCause" below.

NOTE: 64-bit Java versions 1.5, 1.6 and 1.7 are supported, both as your
application and to run the RootCause UI and probe UIs.  A Java runtime
is _not_ included with RootCause. 

Using 64-bit RootCause

If all the applications, including Java, with which you'll be using
RootCause are 64-bit object files, then this product will work just
like the 32-bit version documented in the user's guide:
and the FAQ:

Use the 'file' command to identify whether a binary is 32- or 64-bits.
For example, the following are 64-bit applications that are compatible
with this version of RootCause/Aprobe:

$ file test.exe
test.exe: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

$ file $(which java)
/opt/jdk1.5.0_16/bin/java: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), stripped

If the 'file' output says "32-bit" then the applications are _not_ compatible 
with this version, and you must use the 32-bit version of Aprobe instead.

Using Both 32- and 64-bit RootCause

This 64-bit version of RootCause supports only 64-bit applications, and
requires a 64-bit Java Runtime for the UI.  The RootCause Console can
only be used to instrument 64-bit applications.  However, 64-bit Linux
can also run 32-bit applications, and a separately-installed 32-bit version 
of RootCause may be used to instrument and analyze those.

Generally, it's simplest to use 64- and 32-bit versions from separate
command-line environments that can have one or the other in the local
PATH definition.  However, it is possible to intersperse them within
the same environment, because they use different environment variables:

For compatibility, 32-bit Aprobe continues to use APROBE, APROBE_REGISTRY, 
APROBE_LOG, APROBE_JRE, etc., while 64-bit Aprobe uses APROBE64, APROBE64_REGISTRY, etc.
The one "snag" is PATH, since the programs themselves, such as rootcause and
aprobe, are the same in both.   To work around this, we supply two "wrapper"
scripts, ap32 and ap64.   To use both in a primarily 64-bit environment,
you would do the following:

# set up 32-bit Aprobe environment first
. /opt/aprobe/setup 

# set up 64-bit Aprobe environment (goes at the front of the PATH)
. /opt/aprobe64/setup 

# open a rootcause workspace for your 64-bit application
rootcause open 
rootcause register -x my64bitapp.exe -w my64bitapp.aws
rootcause run my64bitapp.exe 

# do the same for your 32-bit app:
ap32 rootcause open 
ap32 rootcause register -x my32bitapp.exe -w my32bitapp.aws
ap32 rootcause run my32bitapp.exe 

Of course, if you are usually working with a 32-bit application on 
your 64-bit Linux, you can do the opposite:

# set up 64-bit Aprobe environment first
. /opt/aprobe64/setup 

# set up 32-bit Aprobe environment in front
. /opt/aprobe/setup 

# probe a 32-bit application as normal:
aprobe -u trace my32bitapp.exe

# probe a 64-bit app by prefixing the command with ap64:
ap64 aprobe -u trace my64bitapp.exe

Note that this doesn't work with 'rootcause on', but there's another way
to do that:  rootcause on changes the current shell environment by defining
the LD_PRELOAD environment variable.  The definition of this is the same
for both the 32- and 64-bit aprobe installations, so you can just do 
'rootcause on' without the ap32 or ap64 prefix.

However, by default, each installation doesn't know about the other so has
a "stub" for the alternative definition of the libapaudit.so library. You  
can manually connect the 32- and 64-bit installations as follows:

# soft-link 32-bit aprobe's libapaudit into $APROBE64/lib:
cd $APROBE64/lib
mv libapaudit.so libapaudit.so.stub
ln -s $APROBE/lib/libapaudit.so .

# soft-link 64-bit aprobe's libapaudit into $APROBE/lib64:
cd $APROBE/lib64
mv libapaudit.so libapaudit.so.stub
ln -s $APROBE64/lib64/libapaudit.so .

Known Problems and Limitations

 1. RootCause no longer includes a Java Runtime Environment with the
    delivery. In order to use the RootCause Console, you must have a
    64-bit version 1.5 or 1.6 JRE installed.  If this 'java' command is
    not in your path, you can use the APROBE64_JRE environment variable --
    its value is the full path to the 'java' command.
 2. The Aprobe AP call "ap_CallerLocation()" cannot be used iteratively
    to walk up a stack -- it is only valid for the current location.
    You must use the traceback support operations such as the
    ap_LogTraceback macro or the functions that implement it.
 3. The larger size of 64-bit patches means that some locations cannot
    be instrumented, especially on_line.  This may result in warnings
    and incomplete traces in probes that "used to work" on 32-bit

 4. The RootCause Documentation describes only the 32-bit version.
    See "Using 64-bit RootCause" above for the major differences.
    Contact support@ocsystems.com if you have questions.


You may be interested in these features/changes introduced in 
prior releases of RootCause and Aprobe for Linux 64-bit.

Features and Fixes in RootCause 2.3.4b/Aprobe 4.5.4b for (64-bit) Linux
 (2733) apc compile fails on dynamic variant size
 (2735) Gnat 22 variant record issues

Features and Fixes in RootCause 2.3.4a/Aprobe 4.5.4a for (64-bit) Linux
 (2721) statprof -r option produces odd resutls
 (2722) apcgen core dumps reading DWARF variant with no disc
 (2723) Handle Linux reg param for inlined func
 (2727) DWARF2 struct types not represented correctly
 (2729) variant part offset alignment too restrictive
 (2730) memwatch: add option to not track child of fork

Features and Fixes in RootCause 2.3.4/Aprobe 4.5.4 for (64-bit) Linux
 (2688) Apc compiler fails to convert class with incomplete base type
 (2689) Handling of virtual base classes is problematic
 (2691) add probe to log memory for hex dump at format 
 (2692) create a call path profile probe 
 (2694) branch coverage warning about unknown opcode
 (2695) memcheck does not record C++ allocations
 (2699) add AP_MAKE_CPP_STRING to cppstring.h/c
 (2706) traced inlined code line numbers confusing
 (2707) only print Gnat [nnnn] suffix waring in verbose mode
 (2708) bad layout for Ada record with dynamic fields 
 (2711) bad layout for repc spec'd variant type
 (2713) remove help output from coverage output
 (2718) x87 float registers (ST(n)) not correct
 (2719) GNAT 22 exception names have @CXXABI suffix

Features and Fixes in RootCause 2.3.3/Aprobe 4.5.3 for (64-bit) Linux
 (2675) Linux64 doesn't cast param typedef type
 (2678) Create probe to track (large) memory operations
 (2679) hmtb.ual doesn't show tracebacks
 (2681) Allow more flexible call patches (on Linux).
 (2686) Aprobe fails on RHEL8 program wiht dlopen() exceptions

Features and Fixes in RootCause 2.3.2/Aprobe 4.5.2 for (64-bit) Linux
 (2637) Coverage snapshot missing for forked pids
 (2640) statprof report improvements
 (2645) Create apcgen -U type dictionary.
 (2646) union field type not found in apc.
 (2647) hmp report label MaxOutstanding not clear
 (2648) Apcentry can't handle struct with field of stub type
 (2649) create HMTB - heap memory traceback probe
 (2650) Struct field references don't compile in apc
 (2651) prototype using a overloaded function selector
 (2654) ap_ThrowGccString does not work with -q min_heap

Features and Fixes in RootCause 2.3.1/Aprobe 4.5.1 for (64-bit) Linux
 (2577) Linux64 small Gnat return value struct problems
 (2578) Linux Gnat declare blocks may be empty
 (2580) Linux64 core dump processing Ada operator name
 (2581) Linux64 branch coverage does not handle RIP-rel jmp
 (2583) Linux64 Gcc exception info not handled properly
 (2584) Linux64 core dump calling ap_GetExceptionMessage
 (2585) Linux64 char*/char const* exception name wrong
 (2586) Linux64 quietly ignores lines not instrumented
 (2588) Linux64 small Gnat param value struct problems
 (2589) Linux64 ap_DalLibraryLoader not exported
 (2592) COVERFILE should exclude (more) compiler generated func
 (2604) enhancements to memcheck: -S -e -E options
 (2610) Add pragma trap_patch to force a trap patch
 (2613) provide mechanism to register dll/so with Aprobe
 (2614) Linux compiler-generated member names with special char
 (2615) core dump using ap_GnatRaiseException
 (2616) small return type does not set entire reg
 (2617) branch coverage report should show which tests take branch
 (2619) Linux/AIX ap_StubRoutine on_line/offset does not work
 (2620) memcheck should support resetting freed block values
 (2621) Add indication of inlined function to apcgen -D
 (2622) ap_ThrowGccException does not work on Linux
 (2623) Gcc optimized hot/cold funcitons have no debug info
 (2626) apc compile error: Expecting an operand of a numeric type RHEL8
 (2628) track taken test names in brcov
 (2629) stdlib.h compile error using Apcentry on REHL8
 (2630) apc parse error on slice range with Gcc 8
 (2631) fail apc compile of __alignof__ keyword on RHEL8
 (2632) Aprobe fails at startup on RHEL8
 (2633) Gnat exceptions not working with exit probes
 (2634) ap_RaiseGnatException() not working on_line
 (2635) ap_ThrowGccException() not working on_line

Features and Fixes in RootCause 2.3.0/Aprobe 4.5.0 for (64-bit) Linux
 (2552) statprof -a timings off due to bogus extra entry
 (2557) statprof -c fails on linux64
 (2558) COVERFILE should exclude compiler generated functions
 (2559) include date in statprof timestamp
 (2560) APO file doesn't handle single quote for options
 (2561) coverage does not do branches for some functions
 (2562) Linux Gcc try harder to find exception name/message
 (2563) Gnat/Gcc parameters not referenced correctly when -O
 (2564) struct with ap_TimeT fields get false unions
 (2567) x86/64 disassmelber sets float mem ins class OTHER
 (2568) Linux/Gnat Apcentry doesn't allow target type expr outside probe
 (2569) Linux/Gnat dynamically-sized struct fields not generate
 (2570) apcgen core dump processing optimized Gnat code
 (2572) add heap memory profile (hmp) probe
 (2574) Handle Gnat optimized symbol names
 (2575) Gnat optimized code has formals in lexical block

Features and Fixes in RootCause 2.2.9/Aprobe 4.4.9 for (64-bit) Linux
 (2510) brcov.cfg missing from delivery
 (2511) add ability to prune ADI files
 (2512) Apc generates bad layout for variant record
 (2513) apc core dump when emiting warning message
 (2517) merge brcov into coverage and extras
 (2521) add early Aprobe debug
 (2527) default name matching for .adi files not convenient
 (2528) apbc should process similar .brc files
 (2529) memleak and memwatch should have delayed start
 (2531) apcgen reads all debug even if -f used
 (2533) coverage.ual core dumps if -if used
 (2535) apbc doesn't handle similar .brc files with .eab
 (2539) provide option to set statprof interval
 (2540) Upgrades to apcgp.
 (2541) abrmerge should warn if source file not found
 (2545) add statprof option to profile address range
 (2549) Linux params/locals not correct on_line

Features and Fixes in RootCause 2.2.8/Aprobe 4.4.8 for (64-bit) Linux
 (1197) Linux: exit() from thread doesn't trigger program on_exit
 (2490) provide axdln for Linux
 (2492) axdln doesn't tolerate # lines
 (2493) apc rejects -unit name with mixed case
 (2494) apc should prefer local symbol if file scope given
 (2495) Further branch coverage improvements
 (2497) apcgen -T and -M don't work
 (2499) apmkadi on Linux checksum relies on stripped item
 (2501) core dump on stripped Linux executable
 (2505) create a name demangling tool

Features and Fixes in RootCause 2.2.7/Aprobe 4.4.7 for (64-bit) Linux
 (2449) brcov doesn't handle lines from multiple files
 (2451) support enable/disable calls and apdemand in memstat
 (2453) brcov improvements
 (2454) COVERFILE files cause core dump when inserting multiple files
 (2457) apbc.pl doesn't correctly mark some lines
 (2460) apcgen warnings fo C++ symbol _Z
 (2461) apcgen -La reads all debug info
 (2462) apbc/bc.pl does not handle unprobed lines in .bc
 (2463) coverage does not handle fork() well
 (2465) Trap patch on line at end of function not made
 (2472) typo in error message
 (2474) Aprobe setup script must check for all of Aprobe
 (2481) Linux libdal.so does not support preinit
 (2484) -qmin_heap breaks memstat traceback table

Features and Fixes in RootCause 2.2.6d/Aprobe 4.4.6d for (64-bit) Linux
 (2373) wrong filename scope for GNAT symbols
 (2374) $APROBE/setup should update library path
 (2375) Warn if probing system calls 
 (2376) log fixed-length string return value fails
 (2378) partial coverage percentages not obvious
 (2379) Linux target expression problems
 (2382) Linux/Gnat exception reason string not right
 (2383) Add support to raise C++ exception from probe
 (2384) Linux/Gnat position of fixed-length string params wrong
 (2387) emit DWARF warning messages in verbose mode
 (2388) atcmerge should print Aprobe verison
 (2390) support gnatpro 18.0w for APTs
 (2393) support apcgen -D for big Linux executables
 (2394) support wildcard in apcgen -p argument
 (2395) support logging local/global variables in apcgen
 (2396) make ExpandDottedLines the default for coverage
 (2397) support COVERFILE 'sourcefilename' in coverage.cfg
 (2399) fix logging Linux return structs
 (2401) Coverage should report reasons functions no probed
 (2402) Gnat 18.0w DWARF subtypes before base type
 (2404} Warn about mismatched function version in coverage
 (2405) atcmerge should allow merging diff func versions
 (2406) apbc should warn if no source files found/given
 (2407) Warn if Gnat [nnn] suffix not needed
 (2409) atcmerge should produce .bc file as an option
 (2411) Allow aliases of module names (libc.so.6)
 (2412) Linux64 ap_GetTraceback call in mmap probe causes trap
 (2415) Coverage should report maximum offset count for line count
 (2417) Linux64 line number infor missing from dynamic dll
 (2418) Use smaller raw patch on Linux64
 (2423) Add source file matching option to apcgen
 (2424) apcgen generated incorrect scope for func-local var
 (2425) apc can't reference Gnat out params when optimized
 (2426) Problem stubbing abort() with coverage.ual
 (2430) provide access to STL exception message
 (2431) exceptions.ual should provide Ada/C++ exception message string
 (2432) move DWARF warnings to verbose mode
 (2435) Linux/Gnat tag field is hidden whcn logging
 (2436) Linux on_exit probe breaks cross-lang exception raise.
 (2441) memstat statistics are displayed incorrectly
 (2443) provide traceback blacklist address support
 (2445) Linux/Gcc handle cxx11 tag when demangling names
 (2455) apc generated bad code for Gnat in array param ref

Features and Fixes in RootCause 2.2.6c/Aprobe 4.4.6c for 64-bit Linux apps
 (2361) Support for Gnat64 on Linux64
 (2370) wrong symbol filename used due to #include
 (2371) Linux aprobe fails to demangle 'gnat170w9' symbols

Features and Fixes in RootCause 2.2.6b/Aprobe 4.4.6b for 64-bit Linux apps
 (2280) Linux Aprobe problem with LD_PRELOAD in forked process
 (2300) Linux/GNAT upscope references not supported
 (2318) bc support for linux
 (2319) Linux/Gnat exception reason string not reported.
 (2321) sigsegv.ual causes core dump on Linux386 
 (2322) Linux/Gnat Interfaces.C.Long_Float not handled
 (2323) Linux/Gnat records with dynamic sized components layout
 (2324) Aprobe traceback skips a frame when exit probe present
 (2326) Linux apc fails when probes in same file
 (2329) Linux/Gnat cannot instrument lines with extended float instructions
 (2332) Continuation of i2329 Gnat tracebacks
 (2333) Linux (F) 'thread stacks were overlapped'
 (2334) coverage.ual should do final snapshot on abort
 (2335) Linux coverage gets bad line info, causes sigsegv
 (2337) C++ const int declarations not handled
 (2338) ap_RaiseGnatException broken for Gnat shared library
 (2340) implement ap_UnprobeThread to avoid GNAT stackwalk core
 (2341) C++ exceptions not translating correctly
 (2342) GNAT abort task causes core dump unwinding
 (2346) Switch to pthread TLS support for thread tables 
 (2348) Linux/GNAT: on_line probe gets wrong param value in nes
 (2349) use lower case exception name for GnatRaiseExceptionByN
 (2350) support ap_GnatRaiseException in on_entry and on_line

Features and Fixes in RootCause 2.2.6/Aprobe 4.4.6 for 64-bit Linux apps
 1. (2243) Linux apcgen, apc should work with g++/gnat .o files
 2. (2277) Linux atcmerge core dump
 3. (2283) deliver libdal.so.stub for Linux
 4. (2285) Linux support for backtrace(3)
 5. (2286) Linux apcgen assert on char16_t predef type
 6. (2287) Linux/g++ DW_DLE_ATTR_FORM_BAD on huge struct
 7. (2288) Linux/g++ exception class ctor debug info not found
 8. (2290) apc: object location cannot be determined
 9. (2297) apc c constant array treated as Ada
10. (2299) Linux apcgen prints wrong source file name
11. (2310) use MAP_ANONYMOUS on linux

Features and Fixes in RootCause 2.2.5d/Aprobe 4.4.5d for 64-bit Linux apps
 1. (1361) add changes, comments to coverage.2/bc.pl
 2. (1477) coverage sometimes fails to report missed lines
 3. (2203) coverage.cfg should allow inline comments
 4. (2239) Linux/386 bad messages for unknown ins type
 5. (2240) no coverage snapshot on SIGINT or exception
 6. (2241) atcmerge fails merging .tc files from multiple apps
 7. (2242) apcgen -f doesn't handle include files
 8. (2244) apcgen misses C++ members fields in big class
 9. (2246) failure for slice of array'access
10. (2248) core dump on ap_CallerLocation
11. (2249) optionally reset counts between coverage snapshots
12. (2250) each coverage snapshot should write separate .tc and .bc
13. (2253) coverage summary report line should put symbol last
14. (2254) coverage CommaOutputFile gets duplicate content
15. (2256) improvements to coverage reports
17. (2262) support .bc files on Linux
18. (2263) no APC type created for GNAT rep spec'd field
19. (2264) handle private incomplete types in GNAT
20. (2270) report all default options in coverage cfg file

Features and Fixes in RootCause 2.2.5c/Aprobe 4.4.5c for 64-bit Linux apps
 1. (2199) core dumps instrumenting Firefox
 2. (2205) problem parsing gcc 5.x attribute list
 3. (2207) ap_DoCoverageSnapshot() doesn't work
 4. (2215) Linux/Aprobe core dump reading line information 

Features and Fixes in RootCause 2.2.5b/Aprobe 4.4.5b for 64-bit Linux apps
 1. (2131) Linux64 on_line instrumentation crashes
 2. (2138) problem building on centos7
 3. (2139) lines not logged if duplicate timestamp
 4. (2140) linux_amd64.s eh_frame rejected on centos7
 5. (2141) apc parse error parsing complex gnu C type decl
 6. (2144) gcc4.8 requires dwarf4 support
 7. (2160) fix memwatch, memstat examples, output
 8. (2164) java aborts under RC trace

Features and Fixes in 2.2.5/4.4.5

 1. (1921) provide way to avoid NFS apd logging
 2. (2069) unhelpful message for $a[i, j]

Both of these apply to the 32-bit release as well.

Features and Fixes in 2.2.4/4.4.4

 1. (1975) Warn that ap_CallerLocation not fully supported on amd64

Features and Fixes in 2.2.3a/4.4.3a

 1. (1951) apc $7 reference crashes application
 2. (1962) (OCS) don't retry failed demangling
 3. (1964) (OCS) provide libaprobe.adi

Features and Fixes in 2.2.3/4.4.3

 1. (1865) No module found for pa4_get_reraise_ptr(0)
 2. (1880) apcgen: '(E) ap_dwarf_loclist_n: DW_DLE_ATTR_FORM_BAD'
 3. (1885) PROGRAM_ERROR from apc refing C file in Linux PA exe.
 4. (1892) local data addressed wrong on rhel6/gcc4
 5. (1842) Linux mismatched ADI files not flagged
 6. (1884) function inside probe thread hangs
 7. (1896) apc -v shouldn't do gcc -v
 8. (1897) rhel6, handle non-standard frame base reg
 9. (1899) rhel6, Apcentry can't demangle g++ names
10. (1900) gcc4.4, g++ file-static debug info not found
11. (1911) typeof doesn't find C types in Ada binary
12. (1912) redefinition of typedef .. PtrTo_void
13. (1924) aprobe init hangs looping on sig11
14. (1930) provide C++ string helper UAL
15. (1937) 64-bit ROSApp should use APROBE64
16. (1938) linuxamd64 apc generates bad ref to param on_line

Issue 1930 introduces "cppstring.ual" which provides ap_GetCppString() and
ap_SetCppString() functions for use in your probe.  Its usage is 
illustrated in $APROBE/examples/predefined_probes/cppstring/.

All but issues 1924, 1937 and 1938 also apply to the 32-bit release.
More generally, version 2.2.3-64 is the same as (32-bit) 2.2.3 with
these exceptions:

1. It supports only 64-bit Linux native and Java programs.

2. It uses a different mechanism for Java instrumentation which is
   supported only in JVM 1.5 or newer.  

3. The installation location is defined by the environment variable
   APROBE64, and the RootCause user data is defined by the environment
   The value of APROBE64_HOME defaults to $HOME/.rootcause_linux64
   All other product-related environment variables are unchanged.