RootCause User Guide

RootCause Demo

     RCUG 5 RootCause Demo

rcc-8

This demonstration program, included as part of the RootCause installation, has been designed to provide an introduction and overview of the RootCause product. The program is:

$APROBE/demo/RootCause/C++/pi_demo

It is a simple 2-file C++ program, which computes the value of Pi by iteration using multiple threads. You can find the source in the same directory as the executable file.

The goal of this demonstration is to provide an overview of the whole RootCause process, showing initial definition and tuning of the trace, then collection and viewing of more detailed data about a specific function. The demonstration pictured in this chapter was performed on Solaris. Output should be very similar on AIX and Linux.

Set Up

Before running this RootCause Demo, you must install and set up to use RootCause as described in

[["rcc-7.html#MARKER-9-508">Chapter 4, "Getting Started"

Chapter 4, "Getting Started". This will define the APROBE environment variable which is necessary to use RootCause.

In the instructions that follow, we'll use $APROBE to refer to the path where RootCause is installed, for example /opt/aprobe.

Use a Local Disk

We recommend you set your current directory to a disk local to the machine you're running on, though this is not required.

Defined X-Windows DISPLAY

Lastly, make sure your DISPLAY environment variable is set. If you're using a Windows client that is running X emulator software such as eXceed or Reflection, we recommend you move to a Unix display for your initial evaluation. If this is impractical, see

[["rcc-11.html#MARKER-9-1878">"X-emulators: (Exceed, Reflection)"

CHAPTER 8 - RootCause GUI Reference.

Run With RootCause

Run the following commands:

rootcause_on
$APROBE/demo/RootCause/C++/pi_demo
<DIV ID=MARKER-10-714></DIV>rootcause_off

The rootcause_on command enables the automatic logging of every process that is started, and rootcause_off disables this logging. When the pi_demo program is "

[["rcc-6.html#MARKER-9-451">register

CHAPTER 3 - Terminology and Conceptsed" with RootCause, it will be traced according to your specifications as well as simply being logged. The illustration below shows the set up and demo execution..

NOTE: On AIX, the above process is slightly different than shown above. The invocation of the program must be done with the

[["rcc-12.html#MARKER-9-2119">rootcause run

CHAPTER 9 - RootCause Command Reference command directly. For example, the above sequence is changed to:

rootcause run $APROBE/demo/RootCause/C++/pi_demo

See

[["rcc-7.html#MARKER-9-532">"Enabling RootCause for an AIX Application"

"Enabling RootCause for an AIX Application".

View the RootCause Log

Enter the following command:

rootcause open

This will open the RootCause main window, and then a view of the RootCause log. This shows the

[["rcc-11.html#MARKER-9-1725">Trace Display

CHAPTER 8 - RootCause GUI Reference window, the window for viewing all trace events. On the left is the Event Tree; on the upper right is the source/text window; and on the lower right is the "details" window. In the text window you will see some information about the log file.

Locate the APP_START event (in the Trace Event window) associated with the pi_demo program run earlier. To view information about this event, select the APP_START node in the event tree with a left click. This will fill in "details" about that event in the lower right window.With the APP_START pi_demo node highlighted, right-click to bring up the

[["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu

CHAPTER 8 - RootCause GUI Reference.

Click

[["rcc-11.html#MARKER-9-1730">Open Associated Workspace

CHAPTER 8 - RootCause GUI Reference in the popup-menu of the pi_demo APP_START event.

This will open a

[["rcc-11.html#MARKER-9-1434">New Workspace Dialog

CHAPTER 8 - RootCause GUI Reference with the program name and default workspace filled in.

This combination: selecting a node in the tree, then using the popup menu to choose an operation, is the basic way of working within RootCause.

Create a RootCause Workspace

To complete the creation of a RootCause workspace for the pi_demo application:

Click Ok in the New Workspace dialog to complete the creation of the workspace.

Click Yes to confirm that you want to register the pi_demo application with this workspace. This registration is how RootCause determines what applications to trace.

You may close the RootCause Log window opened in a previous step.

You now see the RootCause

[["rcc-11.html#MARKER-9-1321">Workspace Browser

CHAPTER 8 - RootCause GUI Reference. This is described in detail in

[["rcc-11.html#MARKER-9-1320">Chapter 8, "RootCause GUI Reference"

CHAPTER 8 - RootCause GUI Reference.

Define the Trace

There are several aspects to a RootCause trace:

Predefined UALs selected from the [["rcc-11.html#MARKER-9-1323">Workspace Tree
CHAPTER 8 - RootCause GUI Reference;
Method and Line traces selected from the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog
CHAPTER 8 - RootCause GUI Reference;
Probes to gather or preserve data, also selected in the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog
CHAPTER 8 - RootCause GUI Reference; and
User-written custom probes as described in [["rcc-13.html#MARKER-9-2198">"Writing Custom Probes"
CHAPTER 10 - Selected Topics.

In this part of the demo we illustrate the use of Predefined UALs and tracing function calls; then we'll return to the Trace Setup Dialog to add some probes. Custom probes are an advanced feature not presented here.

In the RootCause GUI main window, opened in the previous section:

Enable a UAL

Under the

[["rcc-11.html#MARKER-9-1330">UALs node

CHAPTER 8 - RootCause GUI Reference in the

[["rcc-11.html#MARKER-9-1323">Workspace Tree

CHAPTER 8 - RootCause GUI Reference:

Check the checkbox next to predefined UAL labeled exceptions.ual, as highlighed in the figure above. Simply checking this box will report all user-defined C++ (and Ada) exceptions.

Define Function Traces

Now we'll add traces specific to this application.

Click on the [["rcc-11.html#MARKER-9-1426">Setup
CHAPTER 8 - RootCause GUI Reference button in the button bar.

This will open the

[["rcc-11.html#MARKER-9-1531">Trace Setup Dialog

CHAPTER 8 - RootCause GUI Reference, showing the modules of the application in the

[["rcc-11.html#MARKER-9-1533">Program Contents Tree

CHAPTER 8 - RootCause GUI Reference. The

[["rcc-11.html#MARKER-9-1533">Program Contents Tree

CHAPTER 8 - RootCause GUI Reference identifies the modules, files, functions, methods, and data in the program, and allows you to specify complex actions on each function and method.

For this demo we'll first just specify a trace on the functions in the file pi_demo.cpp, then return later to add data and probes.

Click on the "lever" icon next to the pi_demo module to expand it; then expand the /tmp/demo directory, so you see the nodes for the source files pi.cpp and pi_demo.cpp.
Click on the pi_demo.cpp source file node, then right-click to see the [["rcc-11.html#MARKER-9-1545">Trace Setup Popup Menu
CHAPTER 8 - RootCause GUI Reference.
Click [["rcc-11.html#MARKER-9-1557">Trace All In
CHAPTER 8 - RootCause GUI Reference pi_demo.cpp.
Click the OK button at the bottom right of the dialog to record the trace and dismiss the Trace Setup dialog.

Trace With RootCause

As was done in the preceding section,

[["#MARKER-9-713">"Run With RootCause"

[[]], run the following commands:

rootcause_on
$APROBE/demo/RootCause/C++/pi_demo
rootcause_off

Or, for AIX:

rootcause run $APROBE/demo/RootCause/C++/pi_demo

This time, since the program is registered with a workspace, it will be traced as specified in the workspace, and the resulting output will be recorded within the workspace. There will be some startup delay, but if you notice that the program runs slower once started, this is probably because your workspace is being accessed across the network from your machine. See

[["rcc-13.html#MARKER-9-2164">"RootCause and Efficiency Concerns"

CHAPTER 10 - Selected Topics.

View The Data Index

We're now ready to view the data generated by running with our Trace. This is discussed in detail in

[["rcc-6.html#MARKER-9-366">"RootCause Data Management"

CHAPTER 3 - Terminology and Concepts.

In the

[["rcc-11.html#MARKER-9-1321">Workspace Browser

CHAPTER 8 - RootCause GUI Reference window, do the following:

Click the [["rcc-11.html#MARKER-9-1431">Index
CHAPTER 8 - RootCause GUI Reference button. This will bring up the
[["rcc-11.html#MARKER-9-1680">Trace Index Dialog
CHAPTER 8 - RootCause GUI Reference for the most recently generated data.
In the Trace Index Dialog, click the [["rcc-11.html#MARKER-9-1699">Select Events
CHAPTER 8 - RootCause GUI Reference button.
In the [["rcc-11.html#MARKER-9-1711">Select Events Dialog
CHAPTER 8 - RootCause GUI Reference, check "Exceptions" to index exception events, then click
[["rcc-11.html#MARKER-9-1718">Update
CHAPTER 8 - RootCause GUI Reference to close the dialog and refresh the Index. You should now see something like the dialog below:

In the Trace Index Dialog, double-click on the second item in the table, the first item with an Event name of C++ EXCEPTION.

This will open a Trace Display Dialog centered at that C++ Exception event, as shown below.

Examine and Revise the Trace

The C++ Exception trace event selected from the index should appear highlighted in the Trace Display. This was a result of checking exceptions under

[["#MARKER-9-741">"Enable a UAL"

[[]] above.

The ENTER and EXIT nodes are a result of the "Trace All In pi_demo.exe" action added under

[["#MARKER-9-744">"Define Function Traces"

[[]] above.

You can see more information about the exception in the

[["rcc-11.html#MARKER-9-1830">Event Details Pane

CHAPTER 8 - RootCause GUI Reference in the lower right.

If you scroll to the top of the Event Trace, you should see three threads: one for the main thread, and one for each of the threads which is started to compute Pi.

The event tree is a call tree, and can be very useful. From an ENTER or EXIT node in the tree you can use the

[["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu

CHAPTER 8 - RootCause GUI Reference to:

remove the called function from the set of functions to be traced,
find the next reference to the same function in the trace events, or
go to the function in the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog
CHAPTER 8 - RootCause GUI Reference to trace additional information.

As you step to each event, the

[["rcc-11.html#MARKER-9-1830">Event Details Pane

CHAPTER 8 - RootCause GUI Reference may show additional information about that event. See

[["rcc-11.html#MARKER-9-1725">"Trace Display"

CHAPTER 8 - RootCause GUI Reference for a more complete description of this window.

Take some time to explore the event tree. Then we will look at using the information available here to revise or "tune" the trace used in the next run.

Call Counts

Useful information about the functions called in your program may be obtained by looking at the call frequency as shown in the CALL_COUNTS table.

Select (left-click) the SYN_CALL_COUNTS node, near the end of the event tree.
Right-click (with MB3) to show the [["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu CHAPTER 8 - RootCause GUI Reference on this node.
Click [["rcc-11.html#MARKER-9-1761">Show Associated Table CHAPTER 8 - RootCause GUI Reference. This will open a table listing each called function
and the number of times it was called.
Select (left-click) the extern:"_start()"
entry in the call count table.
Right click to show the popup menu.
Select [["rcc-11.html#MARKER-9-1746">Deselect Function In Trace Setup
CHAPTER 8 - RootCause GUI Reference in the popup menu.

You can also [["rcc-11.html#MARKER-9-1759">Find Function In Trace Events CHAPTER 8 - RootCause GUI Reference to search for functions

in the call tree, and remove them from there.

You can also search for functions

in the call tree, and remove them from there.

When you've finished making changes to the trace, click the Dismiss button at the bottom of the S
YN_
CALL_COUNTS table window, and then
Click the
[["rcc-11.html#MARKER-9-1427">Build CHAPTER 8 - RootCause GUI Reference button in the main window.
Notice the effects of removing these calls in the next trace we generate.

NOTE: In most "real" programs, high-overhead functions selected for tracing are automatically identified and disabled via [["rcc-6.html#MARKER-9-433">load shedding

CHAPTER 3 - Terminology and Concepts, and are listed in the

[["rcc-11.html#MARKER-9-1857">LOAD_SHED Table

CHAPTER 8 - RootCause GUI Reference associated with the LOAD_SHED node at the end of the event tree. This demo doesn't run long enough for the load shedding heuristics to apply. See

[["rcc-6.html#MARKER-9-374">"RootCause Overhead Management" CHAPTER 3 - Terminology and Concepts for a general discussion of load shedding.

Tracing The Details

So far we have seen how the RootCause process works by:

enabling a predefined UAL (
exceptions);
defining a simple trace from the Trace Setup Dialog;
running the application under RootCause;
choosing an event in the Trace Index Dialog;
viewing events in the Trace Display; and
modifying the trace by selecting functions
from the call counts table.

RootCause allows you to record much more than the entry and exit of functions

and threads. You can record data values and insert probes as well.

Open Trace Setup

Click the [["rcc-11.html#MARKER-9-1426">Setup
CHAPTER 8 - RootCause GUI Reference button in the Workspace Browser to return to the
[["rcc-11.html#MARKER-9-1531">Trace Setup Dialog CHAPTER 8 - RootCause GUI Reference.

This is just as in [["#MARKER-9-738">"Define the Trace" [[]] above, but this time we'll record some details about a specific function
,
::ataninvint.

Select A Single Function

Click the "lever" to expand the M pi_demo node to see the Source Files in the module.
Expand the
pi.cpp
node to see the methods.
Select the ataninvint method. This will bring up the source code for this method in the [["rcc-11.html#MARKER-9-1574">Source Pane CHAPTER 8 - RootCause GUI Reference, and show a tree of data to log in ataninvint
in the [["rcc-11.html#MARKER-9-1579">Variables Pane CHAPTER 8 - RootCause GUI Reference.

Log Parameters

Check the box for variable X in the Visible On Entry subtree, and for the return value in the Visible On Exit subtree.

Add a Snapshot Probe

In that same lower-right area, click on the
Probes tab to show the [["rcc-11.html#MARKER-9-1589">Probes Pane
CHAPTER 8 - RootCause GUI Reference. Probes in this context are special
[["rcc-6.html#MARKER-9-396">actions CHAPTER 3 - Terminology and Concepts that can be performed at points in the currently selected function
.
Click the "On" checkbox.
Where it says No Trigger, select Function Entry.
Where it says No Action, select
Log Snapshot.
Where it says ROOTCAUSE_SNAPSHOT, select this and type in "My Snapshot" and hit Enter.

We've now requested that the parameter X

be logged (recorded) on entry to method

ataninvint, and also that a data snapshot be taken at this point and marked with the Event Name "My Snapshot".

Note: A snapshot causes data which might otherwise be deleted do to "data wraparound" to be preserved. In this small demo, a snapshot is not really necessary since it doesn't generate enough data to wrap around and cause old data to be lost. See [["rcc-6.html#MARKER-9-371">"Data Snapshots" CHAPTER 3 - Terminology and Concepts for more information.

Save and Build the Trace

Click the OK button at the bottom right of the dialog to save and build the trace and dismiss the Trace Setup dialog. Note that this will take a bit longer this time because we've created Probe [["rcc-6.html#MARKER-9-396">actions
CHAPTER 3 - Terminology and Concepts that require compilation of an
[["rcc-6.html#MARKER-9-390">APC CHAPTER 3 - Terminology and Concepts file.

Run With RootCause

Again, check that rootcause is enabled with either the [["rcc-12.html#MARKER-9-2094">rootcause_on
CHAPTER 9 - RootCause Command Reference or
[["rcc-12.html#MARKER-9-2129">rootcause status CHAPTER 9 - RootCause Command Reference command. Then run the application by running the pi_demo program
, as described in [["#MARKER-9-759">"Trace With RootCause" [[]].

Index the New Trace

Click the [["rcc-11.html#MARKER-9-1431">Index CHAPTER 8 - RootCause GUI Reference button in the Workspace Browser window.
In the Trace Index Dialog that comes up, double-click on the SNAP entry labeled "My Snapshot" to go right to where our new probes were added.
Select the "ENTER ataninvint" node immediately preceding the Snapshot node to see the value of x
on entry.
Select the corresponding "EXIT ataninvint" node following the Snapshot to see the return value on exit.

Find the Calls Of Interest

The Trace Display window opened from the Trace Index Dialog operation will contain many events, including other calls to ataninvint.

With the "ENTER ataninvint" node selected, right-click to show the [["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu
CHAPTER 8 - RootCause GUI Reference and choose
[["rcc-11.html#MARKER-9-1759">Find Function In Trace Events
CHAPTER 8 - RootCause GUI Reference.
This will bring up the [["rcc-11.html#MARKER-9-1842">Find Text in Trace Events Dialog
CHAPTER 8 - RootCause GUI Reference with the current function name filled in. Click Next, and the next occurrence of this string will be selected. Thereafter you can continue clicking Next, or enter any other string to search for. You can use
[["rcc-11.html#MARKER-9-1763">Find Text in Trace Events
CHAPTER 8 - RootCause GUI Reference from any the popup or Edit menu to search for any string in the current Trace Display.

Where To From Here?

This chapter should have given you a good overview of the process of developing a trace and gathering data for a program. Now you're ready to try it on your own application. The application you use should be a "debug build" so Trace Setup has the information it needs. See

[["rcc-13.html#MARKER-9-2163">"Selected Topics"

CHAPTER 10 - Selected Topics.

Next Previous Index Top