OCS Release Notes Aprobe 4.5.4c Linux32
*READ THIS FIRST*
Release Notes for
RootCause Version 2.3.4c (Aprobe Version 4.5.4c)
for 64-bit Linux Applications
Sep 9, 2022
This file contains brief release notes for RootCause 2.3.4c (Aprobe 4.5.4c)
for 64-bit applications running on x86_64-compatible hardware. Changes
are relative to version 2.3.4b (Aprobe 4.5.4b) dated Feb, 2022.
Features and Fixes in RootCause 2.3.4c/Aprobe 4.5.4c for (64-bit) Linux
(2740) update libocsdwarf for DWARF5 support
Features in prior releases are listed below (look for "PRIOR").
Getting Help
If you have any questions or problems, please contact OC Systems by e-mail at support@ocsystems.com, or by phone at 703-359-8160.
Documentation
This README is currently the only documentation for specific to 64-bit RootCause. Most features and behaviors are as for the 32-bit version. All OC Systems product documentation is available at http://docs.ocsystems.com.
The RootCause Console 'Help' menu points to the OCS Web site, as well as the full documentation delivered with the product.
The RootCause User Guides are available in PDF format in the files RootCauseJava.pdf and RootCauseCpp.pdf in the product installation directory.
The RootCause and Aprobe documentation refers to the APROBE environment variable. When working with a 64-bit installation of Aprobe and RootCause, replace all uses of the APROBE environment variable with APROBE64. The names of the tools themselves (e.g., rootcause, aprobe) are the same in both 32- and 64-bit installations.
Installation
Product Installation and Licensing is described in Chapter 2 of each RootCause User Guide, and in the "install.txt" file located in the root directory of the CD. The installation process is the same as for the 32-bit version, except that there is no prompt for a license key.
After you have successfully installed RootCause, you should work through the Demonstration Program described in Chapter 5 of the RootCause User Guide.
Supported Versions
We have tested with Redhat Enterprise and/or Centos 4, 5, 6, 7.
We fully support programs compiled with gcc and g++ version 3.x, and 4.x compilers. Ada and other languages are not explicitly supported except where their generated code and debug information is common with that for C and C++.
Compatibility
This release is source-code compatible with previous versions, so you shouldn't need to change any APC files. However, this version is NOT binary-compatible, with respect to UAL files. That means you must recompile your probes from their APC source.
NOTE: if you have a 32-bit application running on 64-bit linux, you must use the 32-bit installation of RootCause to build the probes and run your application. See "Using 64-bit RootCause" below.
NOTE: 64-bit Java versions 1.5, 1.6 and 1.7 are supported, both as your application and to run the RootCause UI and probe UIs. A Java runtime is _not_ included with RootCause.
Using 64-bit RootCause
If all the applications, including Java, with which you'll be using RootCause are 64-bit object files, then this product will work just like the 32-bit version documented in the user's guide: http://www.ocsystems.com/user_guide/rootcause/unix/html/index.html and the FAQ: http://www.ocsystems.com/faqs/rc_aprobe_faq.html.
Use the 'file' command to identify whether a binary is 32- or 64-bits. For example, the following are 64-bit applications that are compatible with this version of RootCause/Aprobe:
$ file test.exe test.exe: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped
$ file $(which java) /opt/jdk1.5.0_16/bin/java: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), stripped
If the 'file' output says "32-bit" then the applications are _not_ compatible with this version, and you must use the 32-bit version of Aprobe instead.
Using Both 32- and 64-bit RootCause
This 64-bit version of RootCause supports only 64-bit applications, and requires a 64-bit Java Runtime for the UI. The RootCause Console can only be used to instrument 64-bit applications. However, 64-bit Linux can also run 32-bit applications, and a separately-installed 32-bit version of RootCause may be used to instrument and analyze those.
Generally, it's simplest to use 64- and 32-bit versions from separate command-line environments that can have one or the other in the local PATH definition. However, it is possible to intersperse them within the same environment, because they use different environment variables:
For compatibility, 32-bit Aprobe continues to use APROBE, APROBE_REGISTRY, APROBE_LOG, APROBE_JRE, etc., while 64-bit Aprobe uses APROBE64, APROBE64_REGISTRY, etc. The one "snag" is PATH, since the programs themselves, such as rootcause and aprobe, are the same in both. To work around this, we supply two "wrapper" scripts, ap32 and ap64. To use both in a primarily 64-bit environment, you would do the following:
- set up 32-bit Aprobe environment first
. /opt/aprobe/setup
- set up 64-bit Aprobe environment (goes at the front of the PATH)
. /opt/aprobe64/setup
- open a rootcause workspace for your 64-bit application
rootcause open rootcause register -x my64bitapp.exe -w my64bitapp.aws rootcause run my64bitapp.exe
- do the same for your 32-bit app:
ap32 rootcause open ap32 rootcause register -x my32bitapp.exe -w my32bitapp.aws ap32 rootcause run my32bitapp.exe
Of course, if you are usually working with a 32-bit application on your 64-bit Linux, you can do the opposite:
- set up 64-bit Aprobe environment first
. /opt/aprobe64/setup
- set up 32-bit Aprobe environment in front
. /opt/aprobe/setup
- probe a 32-bit application as normal:
aprobe -u trace my32bitapp.exe
- probe a 64-bit app by prefixing the command with ap64:
ap64 aprobe -u trace my64bitapp.exe
Note that this doesn't work with 'rootcause on', but there's another way to do that: rootcause on changes the current shell environment by defining the LD_PRELOAD environment variable. The definition of this is the same for both the 32- and 64-bit aprobe installations, so you can just do 'rootcause on' without the ap32 or ap64 prefix.
However, by default, each installation doesn't know about the other so has a "stub" for the alternative definition of the libapaudit.so library. You can manually connect the 32- and 64-bit installations as follows:
- soft-link 32-bit aprobe's libapaudit into $APROBE64/lib:
cd $APROBE64/lib mv libapaudit.so libapaudit.so.stub ln -s $APROBE/lib/libapaudit.so .
- soft-link 64-bit aprobe's libapaudit into $APROBE/lib64:
cd $APROBE/lib64 mv libapaudit.so libapaudit.so.stub ln -s $APROBE64/lib64/libapaudit.so .
Known Problems and Limitations
1. RootCause no longer includes a Java Runtime Environment with the delivery. In order to use the RootCause Console, you must have a 64-bit version 1.5 or 1.6 JRE installed. If this 'java' command is not in your path, you can use the APROBE64_JRE environment variable -- its value is the full path to the 'java' command. 2. The Aprobe AP call "ap_CallerLocation()" cannot be used iteratively to walk up a stack -- it is only valid for the current location. You must use the traceback support operations such as the ap_LogTraceback macro or the functions that implement it. 3. The larger size of 64-bit patches means that some locations cannot be instrumented, especially on_line. This may result in warnings and incomplete traces in probes that "used to work" on 32-bit applications.
4. The RootCause Documentation describes only the 32-bit version. See "Using 64-bit RootCause" above for the major differences. Contact support@ocsystems.com if you have questions.
FEATURES INTRODUCED IN PRIOR RELEASES
You may be interested in these features/changes introduced in prior releases of RootCause and Aprobe for Linux 64-bit.
Features and Fixes in RootCause 2.3.4b/Aprobe 4.5.4b for (64-bit) Linux
(2733) apc compile fails on dynamic variant size (2735) Gnat 22 variant record issues
Features and Fixes in RootCause 2.3.4a/Aprobe 4.5.4a for (64-bit) Linux
(2721) statprof -r option produces odd resutls (2722) apcgen core dumps reading DWARF variant with no disc (2723) Handle Linux reg param for inlined func (2727) DWARF2 struct types not represented correctly (2729) variant part offset alignment too restrictive (2730) memwatch: add option to not track child of fork
Features and Fixes in RootCause 2.3.4/Aprobe 4.5.4 for (64-bit) Linux
(2688) Apc compiler fails to convert class with incomplete base type (2689) Handling of virtual base classes is problematic (2691) add probe to log memory for hex dump at format (2692) create a call path profile probe (2694) branch coverage warning about unknown opcode (2695) memcheck does not record C++ allocations (2699) add AP_MAKE_CPP_STRING to cppstring.h/c (2706) traced inlined code line numbers confusing (2707) only print Gnat [nnnn] suffix waring in verbose mode (2708) bad layout for Ada record with dynamic fields (2711) bad layout for repc spec'd variant type (2713) remove help output from coverage output (2718) x87 float registers (ST(n)) not correct (2719) GNAT 22 exception names have @CXXABI suffix
Features and Fixes in RootCause 2.3.3/Aprobe 4.5.3 for (64-bit) Linux
(2675) Linux64 doesn't cast param typedef type (2678) Create probe to track (large) memory operations (2679) hmtb.ual doesn't show tracebacks (2681) Allow more flexible call patches (on Linux). (2686) Aprobe fails on RHEL8 program wiht dlopen() exceptions
Features and Fixes in RootCause 2.3.2/Aprobe 4.5.2 for (64-bit) Linux
(2637) Coverage snapshot missing for forked pids (2640) statprof report improvements (2645) Create apcgen -U type dictionary. (2646) union field type not found in apc. (2647) hmp report label MaxOutstanding not clear (2648) Apcentry can't handle struct with field of stub type (2649) create HMTB - heap memory traceback probe (2650) Struct field references don't compile in apc (2651) prototype using a overloaded function selector (2654) ap_ThrowGccString does not work with -q min_heap
Features and Fixes in RootCause 2.3.1/Aprobe 4.5.1 for (64-bit) Linux
(2577) Linux64 small Gnat return value struct problems (2578) Linux Gnat declare blocks may be empty (2580) Linux64 core dump processing Ada operator name (2581) Linux64 branch coverage does not handle RIP-rel jmp (2583) Linux64 Gcc exception info not handled properly (2584) Linux64 core dump calling ap_GetExceptionMessage (2585) Linux64 char*/char const* exception name wrong (2586) Linux64 quietly ignores lines not instrumented (2588) Linux64 small Gnat param value struct problems (2589) Linux64 ap_DalLibraryLoader not exported (2592) COVERFILE should exclude (more) compiler generated func (2604) enhancements to memcheck: -S -e -E options (2610) Add pragma trap_patch to force a trap patch (2613) provide mechanism to register dll/so with Aprobe (2614) Linux compiler-generated member names with special char (2615) core dump using ap_GnatRaiseException (2616) small return type does not set entire reg (2617) branch coverage report should show which tests take branch (2619) Linux/AIX ap_StubRoutine on_line/offset does not work (2620) memcheck should support resetting freed block values (2621) Add indication of inlined function to apcgen -D (2622) ap_ThrowGccException does not work on Linux (2623) Gcc optimized hot/cold funcitons have no debug info (2626) apc compile error: Expecting an operand of a numeric type RHEL8 (2628) track taken test names in brcov (2629) stdlib.h compile error using Apcentry on REHL8 (2630) apc parse error on slice range with Gcc 8 (2631) fail apc compile of __alignof__ keyword on RHEL8 (2632) Aprobe fails at startup on RHEL8 (2633) Gnat exceptions not working with exit probes (2634) ap_RaiseGnatException() not working on_line (2635) ap_ThrowGccException() not working on_line
Features and Fixes in RootCause 2.3.0/Aprobe 4.5.0 for (64-bit) Linux
(2552) statprof -a timings off due to bogus extra entry (2557) statprof -c fails on linux64 (2558) COVERFILE should exclude compiler generated functions (2559) include date in statprof timestamp (2560) APO file doesn't handle single quote for options (2561) coverage does not do branches for some functions (2562) Linux Gcc try harder to find exception name/message (2563) Gnat/Gcc parameters not referenced correctly when -O (2564) struct with ap_TimeT fields get false unions (2567) x86/64 disassmelber sets float mem ins class OTHER (2568) Linux/Gnat Apcentry doesn't allow target type expr outside probe (2569) Linux/Gnat dynamically-sized struct fields not generate (2570) apcgen core dump processing optimized Gnat code (2572) add heap memory profile (hmp) probe (2574) Handle Gnat optimized symbol names (2575) Gnat optimized code has formals in lexical block
Features and Fixes in RootCause 2.2.9/Aprobe 4.4.9 for (64-bit) Linux
(2510) brcov.cfg missing from delivery (2511) add ability to prune ADI files (2512) Apc generates bad layout for variant record (2513) apc core dump when emiting warning message (2517) merge brcov into coverage and extras (2521) add early Aprobe debug (2527) default name matching for .adi files not convenient (2528) apbc should process similar .brc files (2529) memleak and memwatch should have delayed start (2531) apcgen reads all debug even if -f used (2533) coverage.ual core dumps if -if used (2535) apbc doesn't handle similar .brc files with .eab (2539) provide option to set statprof interval (2540) Upgrades to apcgp. (2541) abrmerge should warn if source file not found (2545) add statprof option to profile address range (2549) Linux params/locals not correct on_line
Features and Fixes in RootCause 2.2.8/Aprobe 4.4.8 for (64-bit) Linux
(1197) Linux: exit() from thread doesn't trigger program on_exit (2490) provide axdln for Linux (2492) axdln doesn't tolerate # lines (2493) apc rejects -unit name with mixed case (2494) apc should prefer local symbol if file scope given (2495) Further branch coverage improvements (2497) apcgen -T and -M don't work (2499) apmkadi on Linux checksum relies on stripped item (2501) core dump on stripped Linux executable (2505) create a name demangling tool
Features and Fixes in RootCause 2.2.7/Aprobe 4.4.7 for (64-bit) Linux
(2449) brcov doesn't handle lines from multiple files (2451) support enable/disable calls and apdemand in memstat (2453) brcov improvements (2454) COVERFILE files cause core dump when inserting multiple files (2457) apbc.pl doesn't correctly mark some lines (2460) apcgen warnings fo C++ symbol _Z (2461) apcgen -La reads all debug info (2462) apbc/bc.pl does not handle unprobed lines in .bc (2463) coverage does not handle fork() well (2465) Trap patch on line at end of function not made (2472) typo in error message (2474) Aprobe setup script must check for all of Aprobe (2481) Linux libdal.so does not support preinit (2484) -qmin_heap breaks memstat traceback table
Features and Fixes in RootCause 2.2.6d/Aprobe 4.4.6d for (64-bit) Linux
(2373) wrong filename scope for GNAT symbols (2374) $APROBE/setup should update library path (2375) Warn if probing system calls (2376) log fixed-length string return value fails (2378) partial coverage percentages not obvious (2379) Linux target expression problems (2382) Linux/Gnat exception reason string not right (2383) Add support to raise C++ exception from probe (2384) Linux/Gnat position of fixed-length string params wrong (2387) emit DWARF warning messages in verbose mode (2388) atcmerge should print Aprobe verison (2390) support gnatpro 18.0w for APTs (2393) support apcgen -D for big Linux executables (2394) support wildcard in apcgen -p argument (2395) support logging local/global variables in apcgen (2396) make ExpandDottedLines the default for coverage (2397) support COVERFILE 'sourcefilename' in coverage.cfg (2399) fix logging Linux return structs (2401) Coverage should report reasons functions no probed (2402) Gnat 18.0w DWARF subtypes before base type (2404} Warn about mismatched function version in coverage (2405) atcmerge should allow merging diff func versions (2406) apbc should warn if no source files found/given (2407) Warn if Gnat [nnn] suffix not needed (2409) atcmerge should produce .bc file as an option (2411) Allow aliases of module names (libc.so.6) (2412) Linux64 ap_GetTraceback call in mmap probe causes trap (2415) Coverage should report maximum offset count for line count (2417) Linux64 line number infor missing from dynamic dll (2418) Use smaller raw patch on Linux64 (2423) Add source file matching option to apcgen (2424) apcgen generated incorrect scope for func-local var (2425) apc can't reference Gnat out params when optimized (2426) Problem stubbing abort() with coverage.ual (2430) provide access to STL exception message (2431) exceptions.ual should provide Ada/C++ exception message string (2432) move DWARF warnings to verbose mode (2435) Linux/Gnat tag field is hidden whcn logging (2436) Linux on_exit probe breaks cross-lang exception raise. (2441) memstat statistics are displayed incorrectly (2443) provide traceback blacklist address support (2445) Linux/Gcc handle cxx11 tag when demangling names (2455) apc generated bad code for Gnat in array param ref
Features and Fixes in RootCause 2.2.6c/Aprobe 4.4.6c for 64-bit Linux apps
(2361) Support for Gnat64 on Linux64 (2370) wrong symbol filename used due to #include (2371) Linux aprobe fails to demangle 'gnat170w9' symbols
Features and Fixes in RootCause 2.2.6b/Aprobe 4.4.6b for 64-bit Linux apps
(2280) Linux Aprobe problem with LD_PRELOAD in forked process (2300) Linux/GNAT upscope references not supported (2318) bc support for linux (2319) Linux/Gnat exception reason string not reported. (2321) sigsegv.ual causes core dump on Linux386 (2322) Linux/Gnat Interfaces.C.Long_Float not handled (2323) Linux/Gnat records with dynamic sized components layout (2324) Aprobe traceback skips a frame when exit probe present (2326) Linux apc fails when probes in same file (2329) Linux/Gnat cannot instrument lines with extended float instructions (2332) Continuation of i2329 Gnat tracebacks (2333) Linux (F) 'thread stacks were overlapped' (2334) coverage.ual should do final snapshot on abort (2335) Linux coverage gets bad line info, causes sigsegv (2337) C++ const int declarations not handled (2338) ap_RaiseGnatException broken for Gnat shared library (2340) implement ap_UnprobeThread to avoid GNAT stackwalk core (2341) C++ exceptions not translating correctly (2342) GNAT abort task causes core dump unwinding (2346) Switch to pthread TLS support for thread tables (2348) Linux/GNAT: on_line probe gets wrong param value in nes (2349) use lower case exception name for GnatRaiseExceptionByN (2350) support ap_GnatRaiseException in on_entry and on_line
Features and Fixes in RootCause 2.2.6/Aprobe 4.4.6 for 64-bit Linux apps
1. (2243) Linux apcgen, apc should work with g++/gnat .o files 2. (2277) Linux atcmerge core dump 3. (2283) deliver libdal.so.stub for Linux 4. (2285) Linux support for backtrace(3) 5. (2286) Linux apcgen assert on char16_t predef type 6. (2287) Linux/g++ DW_DLE_ATTR_FORM_BAD on huge struct 7. (2288) Linux/g++ exception class ctor debug info not found 8. (2290) apc: object location cannot be determined 9. (2297) apc c constant array treated as Ada
10. (2299) Linux apcgen prints wrong source file name 11. (2310) use MAP_ANONYMOUS on linux
Features and Fixes in RootCause 2.2.5d/Aprobe 4.4.5d for 64-bit Linux apps
1. (1361) add changes, comments to coverage.2/bc.pl 2. (1477) coverage sometimes fails to report missed lines 3. (2203) coverage.cfg should allow inline comments 4. (2239) Linux/386 bad messages for unknown ins type 5. (2240) no coverage snapshot on SIGINT or exception 6. (2241) atcmerge fails merging .tc files from multiple apps 7. (2242) apcgen -f doesn't handle include files 8. (2244) apcgen misses C++ members fields in big class 9. (2246) failure for slice of array'access
10. (2248) core dump on ap_CallerLocation 11. (2249) optionally reset counts between coverage snapshots 12. (2250) each coverage snapshot should write separate .tc and .bc 13. (2253) coverage summary report line should put symbol last 14. (2254) coverage CommaOutputFile gets duplicate content 15. (2256) improvements to coverage reports 17. (2262) support .bc files on Linux 18. (2263) no APC type created for GNAT rep spec'd field 19. (2264) handle private incomplete types in GNAT 20. (2270) report all default options in coverage cfg file
Features and Fixes in RootCause 2.2.5c/Aprobe 4.4.5c for 64-bit Linux apps
1. (2199) core dumps instrumenting Firefox 2. (2205) problem parsing gcc 5.x attribute list 3. (2207) ap_DoCoverageSnapshot() doesn't work 4. (2215) Linux/Aprobe core dump reading line information
Features and Fixes in RootCause 2.2.5b/Aprobe 4.4.5b for 64-bit Linux apps
1. (2131) Linux64 on_line instrumentation crashes 2. (2138) problem building on centos7 3. (2139) lines not logged if duplicate timestamp 4. (2140) linux_amd64.s eh_frame rejected on centos7 5. (2141) apc parse error parsing complex gnu C type decl 6. (2144) gcc4.8 requires dwarf4 support 7. (2160) fix memwatch, memstat examples, output 8. (2164) java aborts under RC trace
Features and Fixes in 2.2.5/4.4.5
1. (1921) provide way to avoid NFS apd logging 2. (2069) unhelpful message for $a[i, j]
Both of these apply to the 32-bit release as well.
Features and Fixes in 2.2.4/4.4.4
1. (1975) Warn that ap_CallerLocation not fully supported on amd64
Features and Fixes in 2.2.3a/4.4.3a
1. (1951) apc $7 reference crashes application 2. (1962) (OCS) don't retry failed demangling 3. (1964) (OCS) provide libaprobe.adi
Features and Fixes in 2.2.3/4.4.3
1. (1865) No module found for pa4_get_reraise_ptr(0) 2. (1880) apcgen: '(E) ap_dwarf_loclist_n: DW_DLE_ATTR_FORM_BAD' 3. (1885) PROGRAM_ERROR from apc refing C file in Linux PA exe. 4. (1892) local data addressed wrong on rhel6/gcc4 5. (1842) Linux mismatched ADI files not flagged 6. (1884) function inside probe thread hangs 7. (1896) apc -v shouldn't do gcc -v 8. (1897) rhel6, handle non-standard frame base reg 9. (1899) rhel6, Apcentry can't demangle g++ names
10. (1900) gcc4.4, g++ file-static debug info not found 11. (1911) typeof doesn't find C types in Ada binary 12. (1912) redefinition of typedef .. PtrTo_void 13. (1924) aprobe init hangs looping on sig11 14. (1930) provide C++ string helper UAL 15. (1937) 64-bit ROSApp should use APROBE64 16. (1938) linuxamd64 apc generates bad ref to param on_line
Issue 1930 introduces "cppstring.ual" which provides ap_GetCppString() and ap_SetCppString() functions for use in your probe. Its usage is illustrated in $APROBE/examples/predefined_probes/cppstring/.
All but issues 1924, 1937 and 1938 also apply to the 32-bit release. More generally, version 2.2.3-64 is the same as (32-bit) 2.2.3 with these exceptions:
1. It supports only 64-bit Linux native and Java programs.
2. It uses a different mechanism for Java instrumentation which is
supported only in JVM 1.5 or newer.
3. The installation location is defined by the environment variable
APROBE64, and the RootCause user data is defined by the environment variables APROBE64_REGISTRY, APROBE64_LOG and APROBE64_HOME. The value of APROBE64_HOME defaults to $HOME/.rootcause_linux64 All other product-related environment variables are unchanged.