Difference between revisions of "RCUG 5 RootCause Demo"

From OC Systems Wiki!
Jump to: navigation, search
m
m
Line 1: Line 1:
  
<div><div>
 
  
 
[[RCUG_6_Deploying_The_RootCause_Workspace|Next]] [[RCUG_4_Getting_Started|Previous]] [[RCUG_Index|Index]] [[RCUG_Top|Top]]
 
[[RCUG_6_Deploying_The_RootCause_Workspace|Next]] [[RCUG_4_Getting_Started|Previous]] [[RCUG_Index|Index]] [[RCUG_Top|Top]]
  
 
RootCause User Guide
 
RootCause User Guide
 
</div>
 
  
 
= RootCause Demo =
 
= RootCause Demo =
  
----
+
-<DIV ID="LINK-05demo_rc.fm-firstpage"></DIV><DIV ID=HEADING8-0></DIV>
</div><div><div>
+
<DIV ID="UID-05demo_rc.fm-864080"></DIV>
 
+
<H1>  <DIV ID=MARKER-9-707></DIV> RootCause Demo</H1>
<!-- header -->      {{FULLPAGENAME}}
 
 
 
 
 
 
 
 
 
<DIV>
 
<A NAME=HEADING8></A>
 
 
 
<DIV>
 
<P><A HREF=rcc-9.html>[Next]</A> <A HREF=rcc-7.html>[Previous]</A> <A HREF=rcc-1.html>[Top]</A> <A HREF=rcc-3.html>[Contents]</A> <A HREF=rcc-14.html>[Index]</A></P>
 
<P>RootCause</P>
 
 
 
</DIV>
 
<A NAME="LINK-05demo_rc.fm-firstpage"></A><A NAME=HEADING8-0></A>
 
<A NAME="UID-05demo_rc.fm-864080"></A>
 
<H1>  <A NAME=MARKER-9-707></A> RootCause Demo</H1>
 
 
<HR>
 
<HR>
 
<P> This demonstration program, included as part of the RootCause installation, has been designed to provide an introduction and overview of the RootCause product.  The program is:</P>
 
<P> This demonstration program, included as part of the RootCause installation, has been designed to provide an introduction and overview of the RootCause product.  The program is:</P>
Line 34: Line 15:
 
$APROBE/demo/RootCause/C++/pi_demo
 
$APROBE/demo/RootCause/C++/pi_demo
 
</PRE>
 
</PRE>
<P> <A NAME=MARKER-10-708></A>It is a simple 2-file C++ <A NAME=MARKER-10-709></A>program, which computes the value of Pi by iteration using multiple threads.  You can find the source in the same directory as the executable<A NAME=MARKER-10-710></A> file.  </P>
+
<P> <DIV ID=MARKER-10-708></DIV>It is a simple 2-file C++ <DIV ID=MARKER-10-709></DIV>program, which computes the value of Pi by iteration using multiple threads.  You can find the source in the same directory as the executable<DIV ID=MARKER-10-710></DIV> file.  </P>
 
<P> The goal of this demonstration is to provide an overview of the whole RootCause process, showing initial definition and tuning of the trace, then collection and viewing of more detailed data about a specific function.  The demonstration pictured in this chapter was performed on Solaris.  Output should be very similar on AIX and Linux.</P>
 
<P> The goal of this demonstration is to provide an overview of the whole RootCause process, showing initial definition and tuning of the trace, then collection and viewing of more detailed data about a specific function.  The demonstration pictured in this chapter was performed on Solaris.  Output should be very similar on AIX and Linux.</P>
<A NAME=HEADING8-5></A>
+
<DIV ID=HEADING8-5></DIV>
<A NAME="UID-05demo_rc.fm-956888"></A>
+
<DIV ID="UID-05demo_rc.fm-956888"></DIV>
<H2> <A NAME=MARKER-9-711></A>Set Up</H2>
+
<H2> <DIV ID=MARKER-9-711></DIV>Set Up</H2>
<P> Before running this RootCause Demo, you must install and set up to use RootCause as described in <A HREF="rcc-7.html#MARKER-9-508">Chapter 4, &quot;Getting Started&quot;</A>. This will define the APROBE environment variable which is necessary to use RootCause. </P>
+
<P> Before running this RootCause Demo, you must install and set up to use RootCause as described in [["rcc-7.html#MARKER-9-508">Chapter 4, &quot;Getting Started&quot;</DIV>. This will define the APROBE environment variable which is necessary to use RootCause. </P>
 
<P> In the instructions that follow, we'll use <CODE>$APROBE</CODE> to refer to the path where RootCause is installed, for example <CODE>/opt/aprobe.</CODE></P>
 
<P> In the instructions that follow, we'll use <CODE>$APROBE</CODE> to refer to the path where RootCause is installed, for example <CODE>/opt/aprobe.</CODE></P>
<H3> <A NAME=MARKER-10-712></A>Use a Local Disk</H3>
+
<H3> <DIV ID=MARKER-10-712></DIV>Use a Local Disk</H3>
 
<P> We recommend you set your current directory to a disk local to the machine you're running on, though this is not required.  </P>
 
<P> We recommend you set your current directory to a disk local to the machine you're running on, though this is not required.  </P>
 
<H3> Defined X-Windows DISPLAY</H3>
 
<H3> Defined X-Windows DISPLAY</H3>
<P> Lastly, make sure your DISPLAY environment variable is set.  If you're using a Windows client that is running X emulator software such as eXceed or Reflection, we recommend you move to a Unix display for your initial evaluation.  If this is impractical, see <A HREF="rcc-11.html#MARKER-9-1878">"X-emulators: (Exceed, Reflection)"</A>.</P>
+
<P> Lastly, make sure your DISPLAY environment variable is set.  If you're using a Windows client that is running X emulator software such as eXceed or Reflection, we recommend you move to a Unix display for your initial evaluation.  If this is impractical, see [["rcc-11.html#MARKER-9-1878">"X-emulators: (Exceed, Reflection)"</DIV>.</P>
<A NAME=HEADING8-12></A>
+
<DIV ID=HEADING8-12></DIV>
<A NAME="UID-05demo_rc.fm-956901"></A>
+
<DIV ID="UID-05demo_rc.fm-956901"></DIV>
<H2> <A NAME=MARKER-9-713></A>Run With RootCause</H2>
+
<H2> <DIV ID=MARKER-9-713></DIV>Run With RootCause</H2>
 
<P> Run the following commands:</P>
 
<P> Run the following commands:</P>
 
<PRE>
 
<PRE>
 
rootcause_on
 
rootcause_on
 
$APROBE/demo/RootCause/C++/pi_demo
 
$APROBE/demo/RootCause/C++/pi_demo
<A NAME=MARKER-10-714></A>rootcause_off
+
<DIV ID=MARKER-10-714></DIV>rootcause_off
 
</PRE>
 
</PRE>
<P> The <CODE>rootcause_on</CODE> command enables the automatic logging of every process that is started, and <CODE>rootcause_off</CODE> disables this logging.  When the pi_demo program<A NAME=MARKER-10-715></A> is "<A HREF="rcc-6.html#MARKER-9-451">register</A>ed" with RootCause, it will be traced according to your specifications as well as simply being logged.  The illustration below shows the set up and demo execution..<A NAME=MARKER-10-716></A></P>
+
<P> The <CODE>rootcause_on</CODE> command enables the automatic logging of every process that is started, and <CODE>rootcause_off</CODE> disables this logging.  When the pi_demo program<DIV ID=MARKER-10-715></DIV> is "[["rcc-6.html#MARKER-9-451">register</DIV>ed" with RootCause, it will be traced according to your specifications as well as simply being logged.  The illustration below shows the set up and demo execution..<DIV ID=MARKER-10-716></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_run1.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_run1.gif"></P>
<P> <A NAME=MARKER-10-717></A>NOTE: On AIX, the above process is slightly different than shown above.  The invocation of the program must be done with the <A HREF="rcc-12.html#MARKER-9-2119">rootcause run</A> command directly.  For example, the above sequence is changed to:</P>
+
<P> <DIV ID=MARKER-10-717></DIV>NOTE: On AIX, the above process is slightly different than shown above.  The invocation of the program must be done with the [["rcc-12.html#MARKER-9-2119">rootcause run</DIV> command directly.  For example, the above sequence is changed to:</P>
 
<PRE>
 
<PRE>
 
rootcause run $APROBE/demo/RootCause/C++/pi_demo
 
rootcause run $APROBE/demo/RootCause/C++/pi_demo
 
</PRE>
 
</PRE>
<P> <A NAME=MARKER-10-718></A>See <A HREF="rcc-7.html#MARKER-9-532">&quot;Enabling RootCause for an AIX Application&quot;</A>.</P>
+
<P> <DIV ID=MARKER-10-718></DIV>See [["rcc-7.html#MARKER-9-532">&quot;Enabling RootCause for an AIX Application&quot;</DIV>.</P>
<A NAME=HEADING8-20></A>
+
<DIV ID=HEADING8-20></DIV>
<A NAME="UID-05demo_rc.fm-956926"></A>
+
<DIV ID="UID-05demo_rc.fm-956926"></DIV>
<H2> <A NAME=MARKER-9-719></A>View the RootCause Log</H2>
+
<H2> <DIV ID=MARKER-9-719></DIV>View the RootCause Log</H2>
 
<P> Enter the following command:</P>
 
<P> Enter the following command:</P>
 
<PRE>
 
<PRE>
 
rootcause open
 
rootcause open
 
</PRE>
 
</PRE>
<P> This will open the RootCause main window, and then a view of the RootCause log.  This shows the <EM><A HREF="rcc-11.html#MARKER-9-1725">Trace Display</A></EM> window, the window for viewing all trace events. On the left is the Event Tree; on the upper right is the source/text window; and on the lower right is the "details" window. In the text window you will see some information about the log file. </P>
+
<P> This will open the RootCause main window, and then a view of the RootCause log.  This shows the <EM>[["rcc-11.html#MARKER-9-1725">Trace Display</DIV></EM> window, the window for viewing all trace events. On the left is the Event Tree; on the upper right is the source/text window; and on the lower right is the "details" window. In the text window you will see some information about the log file. </P>
<P> Locate the <A NAME=MARKER-2-720></A>APP_START event (in the Trace Event window) associated with the <CODE>pi_demo</CODE> program<A NAME=MARKER-10-721></A> run earlier. To view information about this event, select the APP_START node in the event tree with a left click<A NAME=MARKER-10-722></A>. This will fill in "details" about that event in the lower right window.With the APP_START pi_demo <A NAME=MARKER-10-723></A>node highlighted, right-click <A NAME=MARKER-10-724></A>to bring up the <A HREF="rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</A>.</P>
+
<P> Locate the <DIV ID=MARKER-2-720></DIV>APP_START event (in the Trace Event window) associated with the <CODE>pi_demo</CODE> program<DIV ID=MARKER-10-721></DIV> run earlier. To view information about this event, select the APP_START node in the event tree with a left click<DIV ID=MARKER-10-722></DIV>. This will fill in "details" about that event in the lower right window.With the APP_START pi_demo <DIV ID=MARKER-10-723></DIV>node highlighted, right-click <DIV ID=MARKER-10-724></DIV>to bring up the [["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</DIV>.</P>
<P> Click <EM><A HREF="rcc-11.html#MARKER-9-1730">Open Associated Workspace</A></EM> in the popup-menu of the pi_demo<A NAME=MARKER-10-725></A> APP_START event.</P>
+
<P> Click <EM>[["rcc-11.html#MARKER-9-1730">Open Associated Workspace</DIV></EM> in the popup-menu of the pi_demo<DIV ID=MARKER-10-725></DIV> APP_START event.</P>
<P> .<A NAME=MARKER-10-726></A></P>
+
<P> .<DIV ID=MARKER-10-726></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_log1.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_log1.gif"></P>
 
<P> </P>
 
<P> </P>
<P> <A NAME=MARKER-10-727></A>This will open a <EM><A HREF="rcc-11.html#MARKER-9-1434">New Workspace Dialog</A></EM> with the program name and default <A NAME=MARKER-2-728></A>workspace filled in.</P>
+
<P> <DIV ID=MARKER-10-727></DIV>This will open a <EM>[["rcc-11.html#MARKER-9-1434">New Workspace Dialog</DIV></EM> with the program name and default <DIV ID=MARKER-2-728></DIV>workspace filled in.</P>
 
<P> This combination: selecting a node in the tree, then using the popup menu to choose an operation, is the basic way of working within RootCause.</P>
 
<P> This combination: selecting a node in the tree, then using the popup menu to choose an operation, is the basic way of working within RootCause.</P>
<A NAME=HEADING8-31></A>
+
<DIV ID=HEADING8-31></DIV>
<A NAME="UID-05demo_rc.fm-959194"></A>
+
<DIV ID="UID-05demo_rc.fm-959194"></DIV>
 
<H2> Create a RootCause Workspace</H2>
 
<H2> Create a RootCause Workspace</H2>
<P> To complete the creation of a RootCause workspace for the <A NAME=MARKER-10-729></A>pi_demo application:</P>
+
<P> To complete the creation of a RootCause workspace for the <DIV ID=MARKER-10-729></DIV>pi_demo application:</P>
 
<OL>
 
<OL>
<LI><P>Click <EM>Ok</EM> in the <EM>New Workspace dialog</EM> to complete the creation of the workspace.<A NAME=MARKER-10-730></A></P>
+
<LI><P>Click <EM>Ok</EM> in the <EM>New Workspace dialog</EM> to complete the creation of the workspace.<DIV ID=MARKER-10-730></DIV></P>
 
</OL>
 
</OL>
<P> <A NAME=MARKER-10-731></A></P>
+
<P> <DIV ID=MARKER-10-731></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_new.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_new.gif"></P>
 
<OL>
 
<OL>
<LI><P>Click <EM>Yes</EM> to confirm that you want to <A NAME=MARKER-2-732></A>register the pi_demo application<A NAME=MARKER-10-733></A> with this workspace. This registration is how RootCause determines what applications to trace.<A NAME=MARKER-10-734></A></P>
+
<LI><P>Click <EM>Yes</EM> to confirm that you want to <DIV ID=MARKER-2-732></DIV>register the pi_demo application<DIV ID=MARKER-10-733></DIV> with this workspace. This registration is how RootCause determines what applications to trace.<DIV ID=MARKER-10-734></DIV></P>
 
</OL>
 
</OL>
 
<P><IMG ALIGN=TOP SRC="sol_not_reg.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_not_reg.gif"></P>
 
<P> You may close the RootCause Log window opened in a previous step.</P>
 
<P> You may close the RootCause Log window opened in a previous step.</P>
<P> You now see the <EM><A NAME=MARKER-2-735></A>RootCause <A HREF="rcc-11.html#MARKER-9-1321">Workspace Browser</A></EM>. This is described in detail in <A HREF="rcc-11.html#MARKER-9-1320">Chapter 8, &quot;RootCause GUI Reference&quot;</A>.<A NAME=MARKER-10-736></A></P>
+
<P> You now see the <EM><DIV ID=MARKER-2-735></DIV>RootCause [["rcc-11.html#MARKER-9-1321">Workspace Browser</DIV></EM>. This is described in detail in [["rcc-11.html#MARKER-9-1320">Chapter 8, &quot;RootCause GUI Reference&quot;</DIV>.<DIV ID=MARKER-10-736></DIV></P>
<P> <A NAME=MARKER-10-737></A></P>
+
<P> <DIV ID=MARKER-10-737></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_wsp.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_wsp.gif"></P>
<A NAME=HEADING8-42></A>
+
<DIV ID=HEADING8-42></DIV>
<A NAME="UID-05demo_rc.fm-956971"></A>
+
<DIV ID="UID-05demo_rc.fm-956971"></DIV>
<H2> <A NAME=MARKER-9-738></A>Define the Trace</H2>
+
<H2> <DIV ID=MARKER-9-738></DIV>Define the Trace</H2>
 
<P> There are several aspects to a RootCause trace:</P>
 
<P> There are several aspects to a RootCause trace:</P>
 
<UL>
 
<UL>
<LI><P>Predefined UALs selected from the <A HREF="rcc-11.html#MARKER-9-1323">Workspace Tree</A>;</P>
+
<LI><P>Predefined UALs selected from the [["rcc-11.html#MARKER-9-1323">Workspace Tree</DIV>;</P>
<LI><P>Method and Line traces selected from the <A HREF="rcc-11.html#MARKER-9-1531">Trace Setup Dialog</A>;</P>
+
<LI><P>Method and Line traces selected from the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog</DIV>;</P>
<LI><P>Probes to gather or preserve data, also selected in the <A HREF="rcc-11.html#MARKER-9-1531">Trace Setup Dialog</A>; and</P>
+
<LI><P>Probes to gather or preserve data, also selected in the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog</DIV>; and</P>
<LI><P>User-written custom probes as described in <A HREF="rcc-13.html#MARKER-9-2198">&quot;Writing Custom Probes&quot;</A>.</P>
+
<LI><P>User-written custom probes as described in [["rcc-13.html#MARKER-9-2198">&quot;Writing Custom Probes&quot;</DIV>.</P>
 
</UL>
 
</UL>
<P> <A NAME=MARKER-10-739></A>In this part of the demo we illustrate the use of Predefined UALs and tracing function<A NAME=MARKER-10-740></A> calls;  then we'll return to the Trace Setup Dialog to add some probes. Custom probes are an advanced feature not presented here.</P>
+
<P> <DIV ID=MARKER-10-739></DIV>In this part of the demo we illustrate the use of Predefined UALs and tracing function<DIV ID=MARKER-10-740></DIV> calls;  then we'll return to the Trace Setup Dialog to add some probes. Custom probes are an advanced feature not presented here.</P>
 
<P> In the RootCause GUI main window, opened in the previous section:</P>
 
<P> In the RootCause GUI main window, opened in the previous section:</P>
<H3> <A NAME=MARKER-9-741></A>Enable a UAL</H3>
+
<H3> <DIV ID=MARKER-9-741></DIV>Enable a UAL</H3>
<P> Under the <A HREF="rcc-11.html#MARKER-9-1330">UALs node</A> in the <A HREF="rcc-11.html#MARKER-9-1323">Workspace Tree</A>:</P>
+
<P> Under the [["rcc-11.html#MARKER-9-1330">UALs node</DIV> in the [["rcc-11.html#MARKER-9-1323">Workspace Tree</DIV>:</P>
 
<OL>
 
<OL>
<LI><P><A NAME=MARKER-10-742></A>Check the checkbox next to predefined UAL labeled <CODE><A NAME=MARKER-2-743></A>exceptions.ual</CODE>, as highlighed in the figure above.  Simply checking this box will report all user-defined C++ (and Ada) exceptions. </P>
+
<LI><P><DIV ID=MARKER-10-742></DIV>Check the checkbox next to predefined UAL labeled <CODE><DIV ID=MARKER-2-743></DIV>exceptions.ual</CODE>, as highlighed in the figure above.  Simply checking this box will report all user-defined C++ (and Ada) exceptions. </P>
 
</OL>
 
</OL>
<H3> <A NAME=MARKER-9-744></A>Define Function<A NAME=MARKER-10-745></A> Traces</H3>
+
<H3> <DIV ID=MARKER-9-744></DIV>Define Function<DIV ID=MARKER-10-745></DIV> Traces</H3>
 
<P> Now we'll add traces specific to this application.</P>
 
<P> Now we'll add traces specific to this application.</P>
 
<OL>
 
<OL>
<LI><P>Click on the <EM><A HREF="rcc-11.html#MARKER-9-1426">Setup</A></EM> button in the button bar. </P>
+
<LI><P>Click on the <EM>[["rcc-11.html#MARKER-9-1426">Setup</DIV></EM> button in the button bar. </P>
 
</OL>
 
</OL>
<P> This will open the <EM><A NAME=MARKER-2-746></A><A HREF="rcc-11.html#MARKER-9-1531">Trace Setup Dialog</A></EM>, showing the modules of the application in the <EM><A NAME=MARKER-2-747></A><A HREF="rcc-11.html#MARKER-9-1533">Program Contents Tree</A></EM>. The <A HREF="rcc-11.html#MARKER-9-1533">Program Contents Tree</A> identifies the <A NAME=MARKER-2-748></A>modules,<A NAME=MARKER-2-749></A> files, <A NAME=MARKER-2-750></A>functions, methods, and <A NAME=MARKER-2-751></A>data <A NAME=MARKER-10-752></A> in the program, and allows you to specify complex actions on each function and method.<A NAME=MARKER-10-753></A></P>
+
<P> This will open the <EM><DIV ID=MARKER-2-746></DIV>[["rcc-11.html#MARKER-9-1531">Trace Setup Dialog</DIV></EM>, showing the modules of the application in the <EM><DIV ID=MARKER-2-747></DIV>[["rcc-11.html#MARKER-9-1533">Program Contents Tree</DIV></EM>. The [["rcc-11.html#MARKER-9-1533">Program Contents Tree</DIV> identifies the <DIV ID=MARKER-2-748></DIV>modules,<DIV ID=MARKER-2-749></DIV> files, <DIV ID=MARKER-2-750></DIV>functions, methods, and <DIV ID=MARKER-2-751></DIV>data <DIV ID=MARKER-10-752></DIV> in the program, and allows you to specify complex actions on each function and method.<DIV ID=MARKER-10-753></DIV></P>
<P> For this demo we'll first just specify a trace on the functions in the file <CODE>pi_demo.cpp</CODE><A NAME=MARKER-10-754></A>, then return later to add data and probes.</P>
+
<P> For this demo we'll first just specify a trace on the functions in the file <CODE>pi_demo.cpp</CODE><DIV ID=MARKER-10-754></DIV>, then return later to add data and probes.</P>
 
<OL>
 
<OL>
<LI><P><A NAME=MARKER-10-755></A>Click on the "lever" icon next to the <CODE>pi_demo</CODE> module to expand it; then expand the <CODE>/tmp/demo</CODE> directory, so you see the nodes for the source files <CODE>pi.cpp</CODE> and <CODE>pi_demo.cpp.</CODE></P>
+
<LI><P><DIV ID=MARKER-10-755></DIV>Click on the "lever" icon next to the <CODE>pi_demo</CODE> module to expand it; then expand the <CODE>/tmp/demo</CODE> directory, so you see the nodes for the source files <CODE>pi.cpp</CODE> and <CODE>pi_demo.cpp.</CODE></P>
<LI><P>Click on the <CODE>pi_demo.cpp</CODE> source file node, then right-click to see the <EM><A HREF="rcc-11.html#MARKER-9-1545">Trace Setup Popup Menu</A></EM>.</P>
+
<LI><P>Click on the <CODE>pi_demo.cpp</CODE> source file node, then right-click to see the <EM>[["rcc-11.html#MARKER-9-1545">Trace Setup Popup Menu</DIV></EM>.</P>
<LI><P>Click <EM><A HREF="rcc-11.html#MARKER-9-1557">Trace All In</A></EM> pi_demo.cpp<A NAME=MARKER-10-756></A>.</P>
+
<LI><P>Click <EM>[["rcc-11.html#MARKER-9-1557">Trace All In</DIV></EM> pi_demo.cpp<DIV ID=MARKER-10-756></DIV>.</P>
 
<LI><P>Click the <EM>OK</EM> button at the bottom right of the dialog to record the trace and dismiss the Trace Setup dialog.</P>
 
<LI><P>Click the <EM>OK</EM> button at the bottom right of the dialog to record the trace and dismiss the Trace Setup dialog.</P>
 
</OL>
 
</OL>
<P> <A NAME=MARKER-10-757></A></P>
+
<P> <DIV ID=MARKER-10-757></DIV></P>
<P> <A NAME=MARKER-10-758></A></P>
+
<P> <DIV ID=MARKER-10-758></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_setup1.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_setup1.gif"></P>
<A NAME=HEADING8-65></A>
+
<DIV ID=HEADING8-65></DIV>
<A NAME="UID-05demo_rc.fm-957001"></A>
+
<DIV ID="UID-05demo_rc.fm-957001"></DIV>
<H2> <A NAME=MARKER-9-759></A>Trace With RootCause</H2>
+
<H2> <DIV ID=MARKER-9-759></DIV>Trace With RootCause</H2>
<P> As was done in the preceding section,<A HREF="#MARKER-9-713">"Run With RootCause"</A>, run the following commands: </P>
+
<P> As was done in the preceding section,[["#MARKER-9-713">"Run With RootCause"</DIV>, run the following commands: </P>
 
<PRE>
 
<PRE>
<A NAME=MARKER-2-760></A>rootcause_on
+
<DIV ID=MARKER-2-760></DIV>rootcause_on
 
$APROBE/demo/RootCause/C++/pi_demo
 
$APROBE/demo/RootCause/C++/pi_demo
<A NAME=MARKER-10-761></A>rootcause_off
+
<DIV ID=MARKER-10-761></DIV>rootcause_off
 
</PRE>
 
</PRE>
<P> Or, for <A NAME=MARKER-2-762></A>AIX: </P>
+
<P> Or, for <DIV ID=MARKER-2-762></DIV>AIX: </P>
 
<PRE>
 
<PRE>
<A NAME=MARKER-2-763></A>rootcause run $APROBE/demo/RootCause/C++/pi_demo
+
<DIV ID=MARKER-2-763></DIV>rootcause run $APROBE/demo/RootCause/C++/pi_demo
 
</PRE>
 
</PRE>
<P> <A NAME=MARKER-10-764></A>This time, since the program is registered with a workspace, it will be traced as specified in the workspace, and the resulting output will be recorded within the workspace. There will be some startup delay, but if you notice that the program runs slower once started, this is probably because your workspace is being accessed across the network from your machine. See <A HREF="rcc-13.html#MARKER-9-2164">"RootCause and Efficiency Concerns"</A>.</P>
+
<P> <DIV ID=MARKER-10-764></DIV>This time, since the program is registered with a workspace, it will be traced as specified in the workspace, and the resulting output will be recorded within the workspace. There will be some startup delay, but if you notice that the program runs slower once started, this is probably because your workspace is being accessed across the network from your machine. See [["rcc-13.html#MARKER-9-2164">"RootCause and Efficiency Concerns"</DIV>.</P>
<A NAME=HEADING8-73></A>
+
<DIV ID=HEADING8-73></DIV>
<A NAME="UID-05demo_rc.fm-957022"></A>
+
<DIV ID="UID-05demo_rc.fm-957022"></DIV>
 
<H2> View The Data Index</H2>
 
<H2> View The Data Index</H2>
<P> We're now ready to view the data generated by running with our Trace.  This is discussed in detail in <A HREF="rcc-6.html#MARKER-9-366">&quot;RootCause Data Management&quot;</A>.</P>
+
<P> We're now ready to view the data generated by running with our Trace.  This is discussed in detail in [["rcc-6.html#MARKER-9-366">&quot;RootCause Data Management&quot;</DIV>.</P>
<P> In the <A HREF="rcc-11.html#MARKER-9-1321">Workspace Browser</A> window, do the following:</P>
+
<P> In the [["rcc-11.html#MARKER-9-1321">Workspace Browser</DIV> window, do the following:</P>
 
<OL>
 
<OL>
<LI><P> Click the <A HREF="rcc-11.html#MARKER-9-1431">Index</A> button.  This will bring up the <EM><A HREF="rcc-11.html#MARKER-9-1680">Trace Index Dialog</A></EM> for the most recently generated data.</P>
+
<LI><P> Click the [["rcc-11.html#MARKER-9-1431">Index</DIV> button.  This will bring up the <EM>[["rcc-11.html#MARKER-9-1680">Trace Index Dialog</DIV></EM> for the most recently generated data.</P>
<LI><P>In the Trace Index Dialog, click the <A HREF="rcc-11.html#MARKER-9-1699">Select Events</A> button.</P>
+
<LI><P>In the Trace Index Dialog, click the [["rcc-11.html#MARKER-9-1699">Select Events</DIV> button.</P>
<LI><P>In the <A HREF="rcc-11.html#MARKER-9-1711">Select Events Dialog</A>, check "Exceptions" to index exception events, then click <A HREF="rcc-11.html#MARKER-9-1718">Update</A> to close the dialog and refresh the Index.  You should now see something like the dialog below:</P>
+
<LI><P>In the [["rcc-11.html#MARKER-9-1711">Select Events Dialog</DIV>, check "Exceptions" to index exception events, then click [["rcc-11.html#MARKER-9-1718">Update</DIV> to close the dialog and refresh the Index.  You should now see something like the dialog below:</P>
 
</OL>
 
</OL>
<P> <A NAME=MARKER-10-765></A></P>
+
<P> <DIV ID=MARKER-10-765></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_index1.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_index1.gif"></P>
 
<OL>
 
<OL>
 
<LI><P>In the Trace Index Dialog, <EM>double-click</EM> on the second item in the table, the first item with an Event name of C++ EXCEPTION.</P>
 
<LI><P>In the Trace Index Dialog, <EM>double-click</EM> on the second item in the table, the first item with an Event name of C++ EXCEPTION.</P>
 
</OL>
 
</OL>
<P> This will open a Trace Display Dialog centered at that <A NAME=MARKER-10-766></A>C++ Exception event, as shown below.</P>
+
<P> This will open a Trace Display Dialog centered at that <DIV ID=MARKER-10-766></DIV>C++ Exception event, as shown below.</P>
<P> <A NAME=MARKER-10-767></A></P>
+
<P> <DIV ID=MARKER-10-767></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_log2.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_log2.gif"></P>
<A NAME=HEADING8-85></A>
+
<DIV ID=HEADING8-85></DIV>
<A NAME="UID-05demo_rc.fm-1059480"></A>
+
<DIV ID="UID-05demo_rc.fm-1059480"></DIV>
 
<H2> Examine and Revise the Trace</H2>
 
<H2> Examine and Revise the Trace</H2>
<P> The C++ Exception <A NAME=MARKER-10-768></A>trace event selected from the index should appear highlighted  in the Trace Display. This was a result of checking <A NAME=MARKER-10-769></A>exceptions under <A HREF="#MARKER-9-741">&quot;Enable a UAL&quot;</A> above.  </P>
+
<P> The C++ Exception <DIV ID=MARKER-10-768></DIV>trace event selected from the index should appear highlighted  in the Trace Display. This was a result of checking <DIV ID=MARKER-10-769></DIV>exceptions under [["#MARKER-9-741">&quot;Enable a UAL&quot;</DIV> above.  </P>
<P> <A NAME=MARKER-10-770></A>  The<A NAME=MARKER-10-771></A> ENTER and EXIT nodes are a result of the "Trace All In <A NAME=MARKER-10-772></A>pi_demo.exe" action added under <A HREF="#MARKER-9-744">&quot;Define Function Traces&quot;</A> above. </P>
+
<P> <DIV ID=MARKER-10-770></DIV>  The<DIV ID=MARKER-10-771></DIV> ENTER and EXIT nodes are a result of the "Trace All In <DIV ID=MARKER-10-772></DIV>pi_demo.exe" action added under [["#MARKER-9-744">&quot;Define Function Traces&quot;</DIV> above. </P>
<P> <A NAME=MARKER-10-773></A>You can see more information about the exception in the <A HREF="rcc-11.html#MARKER-9-1830">Event Details Pane</A> in the lower right<A NAME=MARKER-10-774></A>.</P>
+
<P> <DIV ID=MARKER-10-773></DIV>You can see more information about the exception in the [["rcc-11.html#MARKER-9-1830">Event Details Pane</DIV> in the lower right<DIV ID=MARKER-10-774></DIV>.</P>
 
<P> If you scroll to the top of the Event Trace, you should see three threads: one for the main thread, and one for each of the threads which is started to compute Pi.</P>
 
<P> If you scroll to the top of the Event Trace, you should see three threads: one for the main thread, and one for each of the threads which is started to compute Pi.</P>
<P> <A NAME=MARKER-10-775></A>The <A NAME=MARKER-2-776></A>event tree is a <A NAME=MARKER-2-777></A>call tree, and can be very useful. From an <A NAME=MARKER-2-778></A>ENTER or <A NAME=MARKER-2-779></A>EXIT  node in the tree you can use the <EM><A HREF="rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</A></EM> to:</P>
+
<P> <DIV ID=MARKER-10-775></DIV>The <DIV ID=MARKER-2-776></DIV>event tree is a <DIV ID=MARKER-2-777></DIV>call tree, and can be very useful. From an <DIV ID=MARKER-2-778></DIV>ENTER or <DIV ID=MARKER-2-779></DIV>EXIT  node in the tree you can use the <EM>[["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</DIV></EM> to:</P>
 
<UL>
 
<UL>
<LI><P>remove the called function<A NAME=MARKER-10-780></A> from the set of function<A NAME=MARKER-10-781></A>s to be traced, </P>
+
<LI><P>remove the called function<DIV ID=MARKER-10-780></DIV> from the set of function<DIV ID=MARKER-10-781></DIV>s to be traced, </P>
<LI><P>find the next reference to the same function<A NAME=MARKER-10-782></A> in the trace events, or </P>
+
<LI><P>find the next reference to the same function<DIV ID=MARKER-10-782></DIV> in the trace events, or </P>
<LI><P>go to the function<A NAME=MARKER-10-783></A> in the <EM><A NAME=MARKER-2-784></A><A HREF="rcc-11.html#MARKER-9-1531">Trace Setup Dialog</A></EM> to trace additional information. </P>
+
<LI><P>go to the function<DIV ID=MARKER-10-783></DIV> in the <EM><DIV ID=MARKER-2-784></DIV>[["rcc-11.html#MARKER-9-1531">Trace Setup Dialog</DIV></EM> to trace additional information. </P>
 
</UL>
 
</UL>
<P> As you step to each event, the <EM><A HREF="rcc-11.html#MARKER-9-1830">Event Details Pane</A></EM> may show additional information about that event. See <A HREF="rcc-11.html#MARKER-9-1725">&quot;Trace Display&quot;</A> for a more complete description of this window.</P>
+
<P> As you step to each event, the <EM>[["rcc-11.html#MARKER-9-1830">Event Details Pane</DIV></EM> may show additional information about that event. See [["rcc-11.html#MARKER-9-1725">&quot;Trace Display&quot;</DIV> for a more complete description of this window.</P>
<P> Take some time to explore the event tree.  Then we will look at using the information available here to revise or " <A NAME=MARKER-2-785></A>tune" the trace used in the next run.</P>
+
<P> Take some time to explore the event tree.  Then we will look at using the information available here to revise or " <DIV ID=MARKER-2-785></DIV>tune" the trace used in the next run.</P>
 
<H3> Call Counts</H3>
 
<H3> Call Counts</H3>
<P> Useful information about the function<A NAME=MARKER-10-786></A>s called in your program may be obtained by looking at the call frequency as shown in the CALL_COUNTS table.</P>
+
<P> Useful information about the function<DIV ID=MARKER-10-786></DIV>s called in your program may be obtained by looking at the call frequency as shown in the CALL_COUNTS table.</P>
 
<OL>
 
<OL>
<LI><P>Select (left-click) the S<A NAME=MARKER-2-787></A>YN_<A NAME=MARKER-10-788></A>CALL_COUNTS node, near the end of the event tree.  </P>
+
<LI><P>Select (left-click) the S<DIV ID=MARKER-2-787></DIV>YN_<DIV ID=MARKER-10-788></DIV>CALL_COUNTS node, near the end of the event tree.  </P>
<LI><P>Right-click (with MB3) to show the <EM><A HREF="rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</A></EM> on this node.</P>
+
<LI><P>Right-click (with MB3) to show the <EM>[["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</DIV></EM> on this node.</P>
<LI><P>Click <EM><A HREF="rcc-11.html#MARKER-9-1761">Show Associated Table</A></EM>. This will open a table listing each called function<A NAME=MARKER-10-789></A> and the number of times it was called.</P>
+
<LI><P>Click <EM>[["rcc-11.html#MARKER-9-1761">Show Associated Table</DIV></EM>. This will open a table listing each called function<DIV ID=MARKER-10-789></DIV> and the number of times it was called.</P>
<LI><P>Select (left-click) the <CODE>extern:&quot;_start()&quot;<A NAME=MARKER-10-790></A></CODE> entry in the call count table.</P>
+
<LI><P>Select (left-click) the <CODE>extern:&quot;_start()&quot;<DIV ID=MARKER-10-790></DIV></CODE> entry in the call count table.</P>
 
<LI><P> Right click to show the popup menu.</P>
 
<LI><P> Right click to show the popup menu.</P>
<LI><P>Select <EM><A HREF="rcc-11.html#MARKER-9-1746">Deselect Function In Trace Setup</A></EM> in the popup menu.</P>
+
<LI><P>Select <EM>[["rcc-11.html#MARKER-9-1746">Deselect Function In Trace Setup</DIV></EM> in the popup menu.</P>
 
</OL>
 
</OL>
<P> You can also <EM><A HREF="rcc-11.html#MARKER-9-1759">Find Function In Trace Events</A></EM> to search for functions<A NAME=MARKER-10-791></A> in the call tree, and remove them from there.</P>
+
<P> You can also <EM>[["rcc-11.html#MARKER-9-1759">Find Function In Trace Events</DIV></EM> to search for functions<DIV ID=MARKER-10-791></DIV> in the call tree, and remove them from there.</P>
<P> You can also search for functions<A NAME=MARKER-10-792></A> in the call tree, and remove them from there.<A NAME=MARKER-10-793></A></P>
+
<P> You can also search for functions<DIV ID=MARKER-10-792></DIV> in the call tree, and remove them from there.<DIV ID=MARKER-10-793></DIV></P>
<P> <A NAME=MARKER-10-794></A></P>
+
<P> <DIV ID=MARKER-10-794></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_calls.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_calls.gif"></P>
 
<OL>
 
<OL>
<LI><P>When you've finished making changes to the trace, click the <EM>Dismiss</EM> button at the bottom of the  S<A NAME=MARKER-2-795></A>YN_<A NAME=MARKER-10-796></A>CALL_COUNTS table window, and then </P>
+
<LI><P>When you've finished making changes to the trace, click the <EM>Dismiss</EM> button at the bottom of the  S<DIV ID=MARKER-2-795></DIV>YN_<DIV ID=MARKER-10-796></DIV>CALL_COUNTS table window, and then </P>
<LI><P>Click the <EM><A NAME=MARKER-2-797></A><A HREF="rcc-11.html#MARKER-9-1427">Build</A></EM> button in the main window.</P>
+
<LI><P>Click the <EM><DIV ID=MARKER-2-797></DIV>[["rcc-11.html#MARKER-9-1427">Build</DIV></EM> button in the main window.</P>
 
<LI><P>Notice the effects of removing these calls in the next trace we generate.</P>
 
<LI><P>Notice the effects of removing these calls in the next trace we generate.</P>
 
</OL>
 
</OL>
<P> NOTE: In most "real" programs, high-overhead functions selected for tracing are automatically identified and disabled via <A HREF="rcc-6.html#MARKER-9-433">load shedding</A>, and are listed in the <A HREF="rcc-11.html#MARKER-9-1857">LOAD_SHED Table</A> associated with the LOAD_SHED node at the end of the event tree.  This demo doesn't run long enough for the load shedding heuristics to apply.  See <A HREF="rcc-6.html#MARKER-9-374">&quot;RootCause Overhead Management&quot;</A> for a general discussion of load shedding.</P>
+
<P> NOTE: In most "real" programs, high-overhead functions selected for tracing are automatically identified and disabled via [["rcc-6.html#MARKER-9-433">load shedding</DIV>, and are listed in the [["rcc-11.html#MARKER-9-1857">LOAD_SHED Table</DIV> associated with the LOAD_SHED node at the end of the event tree.  This demo doesn't run long enough for the load shedding heuristics to apply.  See [["rcc-6.html#MARKER-9-374">&quot;RootCause Overhead Management&quot;</DIV> for a general discussion of load shedding.</P>
<A NAME=HEADING8-112></A>
+
<DIV ID=HEADING8-112></DIV>
<A NAME="UID-05demo_rc.fm-972570"></A>
+
<DIV ID="UID-05demo_rc.fm-972570"></DIV>
 
<H2> Tracing The Details</H2>
 
<H2> Tracing The Details</H2>
 
<P> So far we have seen how the RootCause process works by:</P>
 
<P> So far we have seen how the RootCause process works by:</P>
 
<UL>
 
<UL>
<LI><P>enabling a predefined UAL (<A NAME=MARKER-10-798></A>exceptions);</P>
+
<LI><P>enabling a predefined UAL (<DIV ID=MARKER-10-798></DIV>exceptions);</P>
 
<LI><P>defining a simple trace from the Trace Setup Dialog; </P>
 
<LI><P>defining a simple trace from the Trace Setup Dialog; </P>
 
<LI><P>running the application under RootCause; </P>
 
<LI><P>running the application under RootCause; </P>
 
<LI><P>choosing an event in the Trace Index Dialog;</P>
 
<LI><P>choosing an event in the Trace Index Dialog;</P>
 
<LI><P>viewing events in the Trace Display; and</P>
 
<LI><P>viewing events in the Trace Display; and</P>
<LI><P>modifying the trace by selecting functions<A NAME=MARKER-10-799></A> from the call counts table.</P>
+
<LI><P>modifying the trace by selecting functions<DIV ID=MARKER-10-799></DIV> from the call counts table.</P>
 
</UL>
 
</UL>
<P> RootCause allows you to record much more than the entry and exit of functions<A NAME=MARKER-10-800></A> and threads.  You can record data values and insert probes as well.</P>
+
<P> RootCause allows you to record much more than the entry and exit of functions<DIV ID=MARKER-10-800></DIV> and threads.  You can record data values and insert probes as well.</P>
 
<H3> Open Trace Setup</H3>
 
<H3> Open Trace Setup</H3>
 
<OL>
 
<OL>
<LI><P>Click the <A HREF="rcc-11.html#MARKER-9-1426">Setup</A> button in the Workspace Browser to return to the <A HREF="rcc-11.html#MARKER-9-1531">Trace Setup Dialog</A>. </P>
+
<LI><P>Click the [["rcc-11.html#MARKER-9-1426">Setup</DIV> button in the Workspace Browser to return to the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog</DIV>. </P>
<P>This is just as in <A HREF="#MARKER-9-738">"Define the Trace"</A> above, but this time we'll record some details about a specific function<A NAME=MARKER-10-801></A>, <A NAME=MARKER-10-802></A><CODE>::ataninvint<A NAME=MARKER-10-803></A></CODE>.</P>
+
<P>This is just as in [["#MARKER-9-738">"Define the Trace"</DIV> above, but this time we'll record some details about a specific function<DIV ID=MARKER-10-801></DIV>, <DIV ID=MARKER-10-802></DIV><CODE>::ataninvint<DIV ID=MARKER-10-803></DIV></CODE>.</P>
 
</OL>
 
</OL>
<H3> Select A Single Function<A NAME=MARKER-10-804></A></H3>
+
<H3> Select A Single Function<DIV ID=MARKER-10-804></DIV></H3>
 
<OL>
 
<OL>
 
<LI><P>Click the "lever" to expand the <B>M </B>pi_demo  node to see the Source Files in the module.</P>
 
<LI><P>Click the "lever" to expand the <B>M </B>pi_demo  node to see the Source Files in the module.</P>
<LI><P>Expand the <A NAME=MARKER-2-805></A>pi.cpp<A NAME=MARKER-2-806></A><A NAME=MARKER-10-807></A> node to see the methods.</P>
+
<LI><P>Expand the <DIV ID=MARKER-2-805></DIV>pi.cpp<DIV ID=MARKER-2-806></DIV><DIV ID=MARKER-10-807></DIV> node to see the methods.</P>
<LI><P>Select the <CODE>ataninvint</CODE> method. This will bring up the source code for this method in the <A HREF="rcc-11.html#MARKER-9-1574">Source Pane</A>, and show a tree of data to log in <CODE>ataninvint </CODE><A NAME=MARKER-10-808></A>in the <EM><A HREF="rcc-11.html#MARKER-9-1579">Variables Pane</A></EM>.</P>
+
<LI><P>Select the <CODE>ataninvint</CODE> method. This will bring up the source code for this method in the [["rcc-11.html#MARKER-9-1574">Source Pane</DIV>, and show a tree of data to log in <CODE>ataninvint </CODE><DIV ID=MARKER-10-808></DIV>in the <EM>[["rcc-11.html#MARKER-9-1579">Variables Pane</DIV></EM>.</P>
 
</OL>
 
</OL>
 
<H3> Log Parameters</H3>
 
<H3> Log Parameters</H3>
 
<OL>
 
<OL>
<LI><P>Check the box for variable X in the <EM>Visible On Entry</EM> subtree, and for the return value in the <EM>Visible On Exit</EM> subtree.<EM><A NAME=MARKER-10-809></A></EM></P>
+
<LI><P>Check the box for variable X in the <EM>Visible On Entry</EM> subtree, and for the return value in the <EM>Visible On Exit</EM> subtree.<EM><DIV ID=MARKER-10-809></DIV></EM></P>
 
</OL>
 
</OL>
<P> <A NAME=MARKER-10-810></A></P>
+
<P> <DIV ID=MARKER-10-810></DIV></P>
<P> <A NAME=MARKER-10-811></A></P>
+
<P> <DIV ID=MARKER-10-811></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_vars.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_vars.gif"></P>
<H3> <A NAME=MARKER-9-812></A>Add a Snapshot Probe</H3>
+
<H3> <DIV ID=MARKER-9-812></DIV>Add a Snapshot Probe</H3>
 
<OL>
 
<OL>
<LI><P>In that same lower-right area, click on the <EM><A NAME=MARKER-2-813></A>Probes</EM> tab to show the <EM><A HREF="rcc-11.html#MARKER-9-1589">Probes Pane</A></EM>. Probes in this context are special <A HREF="rcc-6.html#MARKER-9-396">actions</A> that can be performed at points in the currently selected function<A NAME=MARKER-10-814></A>. </P>
+
<LI><P>In that same lower-right area, click on the <EM><DIV ID=MARKER-2-813></DIV>Probes</EM> tab to show the <EM>[["rcc-11.html#MARKER-9-1589">Probes Pane</DIV></EM>. Probes in this context are special [["rcc-6.html#MARKER-9-396">actions</DIV> that can be performed at points in the currently selected function<DIV ID=MARKER-10-814></DIV>. </P>
 
<LI><P>Click the "On" checkbox.</P>
 
<LI><P>Click the "On" checkbox.</P>
 
<LI><P>Where it says No Trigger, select Function Entry.</P>
 
<LI><P>Where it says No Trigger, select Function Entry.</P>
<LI><P>Where it says No Action, select <A NAME=MARKER-2-815></A>Log Snapshot.</P>
+
<LI><P>Where it says No Action, select <DIV ID=MARKER-2-815></DIV>Log Snapshot.</P>
 
<LI><P>Where it says ROOTCAUSE_SNAPSHOT, select this and type in "My Snapshot" and hit Enter.</P>
 
<LI><P>Where it says ROOTCAUSE_SNAPSHOT, select this and type in "My Snapshot" and hit Enter.</P>
 
</OL>
 
</OL>
<P> <A NAME=MARKER-10-816></A></P>
+
<P> <DIV ID=MARKER-10-816></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_probes.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_probes.gif"></P>
<P> We've now requested that the parameter X <A NAME=MARKER-10-817></A>be logged (recorded) on entry to method <A NAME=MARKER-10-818></A><CODE>ataninvint</CODE>, and also that a data snapshot be taken at this point and marked with the Event Name "My Snapshot".    </P>
+
<P> We've now requested that the parameter X <DIV ID=MARKER-10-817></DIV>be logged (recorded) on entry to method <DIV ID=MARKER-10-818></DIV><CODE>ataninvint</CODE>, and also that a data snapshot be taken at this point and marked with the Event Name "My Snapshot".    </P>
<P> <EM>Note</EM>:  A snapshot causes data which might otherwise be deleted do to "data wraparound" to be preserved.  In this small demo, a snapshot is not really necessary since it doesn't generate enough data to wrap around and cause old data to be lost.  See <A HREF="rcc-6.html#MARKER-9-371">&quot;Data Snapshots&quot;</A> for more information.</P>
+
<P> <EM>Note</EM>:  A snapshot causes data which might otherwise be deleted do to "data wraparound" to be preserved.  In this small demo, a snapshot is not really necessary since it doesn't generate enough data to wrap around and cause old data to be lost.  See [["rcc-6.html#MARKER-9-371">&quot;Data Snapshots&quot;</DIV> for more information.</P>
 
<H3> Save and Build the Trace</H3>
 
<H3> Save and Build the Trace</H3>
 
<OL>
 
<OL>
<LI><P>Click the <EM>OK</EM> button at the bottom right of the dialog to save and build the trace and dismiss the Trace Setup dialog. Note that this will take a bit longer this time because we've created Probe <A HREF="rcc-6.html#MARKER-9-396">actions</A> that require compilation of an <A HREF="rcc-6.html#MARKER-9-390">APC</A> file.</P>
+
<LI><P>Click the <EM>OK</EM> button at the bottom right of the dialog to save and build the trace and dismiss the Trace Setup dialog. Note that this will take a bit longer this time because we've created Probe [["rcc-6.html#MARKER-9-396">actions</DIV> that require compilation of an [["rcc-6.html#MARKER-9-390">APC</DIV> file.</P>
 
</OL>
 
</OL>
<H3> <A NAME=MARKER-9-819></A>Run With RootCause</H3>
+
<H3> <DIV ID=MARKER-9-819></DIV>Run With RootCause</H3>
 
<OL>
 
<OL>
<LI><P>Again, check that  rootcause is enabled with either the <A HREF="rcc-12.html#MARKER-9-2094">rootcause_on</A> or <A HREF="rcc-12.html#MARKER-9-2129">rootcause status</A> command.  Then run the application by running the pi_demo program<A NAME=MARKER-10-820></A>, as described in <A HREF="#MARKER-9-759">"Trace With RootCause"</A>.</P>
+
<LI><P>Again, check that  rootcause is enabled with either the [["rcc-12.html#MARKER-9-2094">rootcause_on</DIV> or [["rcc-12.html#MARKER-9-2129">rootcause status</DIV> command.  Then run the application by running the pi_demo program<DIV ID=MARKER-10-820></DIV>, as described in [["#MARKER-9-759">"Trace With RootCause"</DIV>.</P>
 
</OL>
 
</OL>
 
<H3> Index the New Trace</H3>
 
<H3> Index the New Trace</H3>
 
<OL>
 
<OL>
<LI><P>Click the <EM><A HREF="rcc-11.html#MARKER-9-1431">Index</A></EM> button in the Workspace Browser window. </P>
+
<LI><P>Click the <EM>[["rcc-11.html#MARKER-9-1431">Index</DIV></EM> button in the Workspace Browser window. </P>
 
<LI><P>In the Trace Index Dialog that comes up, double-click on the SNAP entry labeled "My Snapshot" to go right to where our new probes were added.</P>
 
<LI><P>In the Trace Index Dialog that comes up, double-click on the SNAP entry labeled "My Snapshot" to go right to where our new probes were added.</P>
<LI><P>Select the "ENTER ataninvint" node immediately preceding the Snapshot node to see the value of x <A NAME=MARKER-10-821></A>on entry.</P>
+
<LI><P>Select the "ENTER ataninvint" node immediately preceding the Snapshot node to see the value of x <DIV ID=MARKER-10-821></DIV>on entry.</P>
 
<LI><P>Select the corresponding "EXIT ataninvint" node following the Snapshot to see the return value on exit.</P>
 
<LI><P>Select the corresponding "EXIT ataninvint" node following the Snapshot to see the return value on exit.</P>
 
</OL>
 
</OL>
Line 254: Line 235:
 
<P> The Trace Display window opened from the Trace Index Dialog operation will contain many events, including other calls to <CODE>ataninvint</CODE>.</P>
 
<P> The Trace Display window opened from the Trace Index Dialog operation will contain many events, including other calls to <CODE>ataninvint</CODE>.</P>
 
<OL>
 
<OL>
<LI><P>With the "ENTER ataninvint" node selected, right-click to show the <EM><A HREF="rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</A> and choose <A HREF="rcc-11.html#MARKER-9-1759">Find Function In Trace Events</A>.</EM></P>
+
<LI><P>With the "ENTER ataninvint" node selected, right-click to show the <EM>[["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu</DIV> and choose [["rcc-11.html#MARKER-9-1759">Find Function In Trace Events</DIV>.</EM></P>
<LI><P>This will bring up the <EM><A HREF="rcc-11.html#MARKER-9-1842">Find Text in Trace Events Dialog</A></EM> with the current function<A NAME=MARKER-10-822></A> name filled in.  Click Next, and the next occurrence of this string will be selected.  Thereafter you can continue clicking <EM>Next</EM>, or enter any other string to search for.  You can use  <EM><A HREF="rcc-11.html#MARKER-9-1763">Find Text in Trace Events</A></EM> from any the popup or Edit menu to search for any string in the current Trace Display.</P>
+
<LI><P>This will bring up the <EM>[["rcc-11.html#MARKER-9-1842">Find Text in Trace Events Dialog</DIV></EM> with the current function<DIV ID=MARKER-10-822></DIV> name filled in.  Click Next, and the next occurrence of this string will be selected.  Thereafter you can continue clicking <EM>Next</EM>, or enter any other string to search for.  You can use  <EM>[["rcc-11.html#MARKER-9-1763">Find Text in Trace Events</DIV></EM> from any the popup or Edit menu to search for any string in the current Trace Display.</P>
 
</OL>
 
</OL>
<P> <A NAME=MARKER-10-823></A></P>
+
<P> <DIV ID=MARKER-10-823></DIV></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_log3.gif"></P>
 
<P><IMG ALIGN=TOP SRC="sol_cpp_log3.gif"></P>
<A NAME=HEADING8-158></A>
+
<DIV ID=HEADING8-158></DIV>
<A NAME="UID-05demo_rc.fm-1109014"></A>
+
<DIV ID="UID-05demo_rc.fm-1109014"></DIV>
 
<H2> Where To From Here?</H2>
 
<H2> Where To From Here?</H2>
<P> This chapter should have given you a good overview of the process of developing a trace and gathering data for a program. Now you're ready to try it on your own application. The application you use should be a "debug build" so Trace Setup has the information it needs. See <A HREF="rcc-13.html#MARKER-9-2163">&quot;Selected Topics&quot;</A>.</P>
+
<P> This chapter should have given you a good overview of the process of developing a trace and gathering data for a program. Now you're ready to try it on your own application. The application you use should be a "debug build" so Trace Setup has the information it needs. See [["rcc-13.html#MARKER-9-2163">&quot;Selected Topics&quot;</DIV>.</P>
<A NAME="LINK-05demo_rc.fm-lastpage"></A>
+
<DIV ID="LINK-05demo_rc.fm-lastpage"></DIV>
 
</DIV>
 
</DIV>
  
 
<DIV>
 
<DIV>
 
<HR>
 
<HR>
<P><A HREF=rcc-9.html>[Next]</A> <A HREF=rcc-7.html>[Previous]</A> <A HREF=rcc-1.html>[Top]</A> <A HREF=rcc-3.html>[Contents]</A> <A HREF=rcc-14.html>[Index]</A></P>
+
<P>[[rcc-9.html>[Next]</DIV> [[rcc-7.html>[Previous]</DIV> [[rcc-1.html>[Top]</DIV> [[rcc-3.html>[Contents]</DIV> [[rcc-14.html>[Index]</DIV></P>
 
<P></P>
 
<P></P>
 
<ADDRESS>Copyright 2006  OC Systems, Inc.</ADDRESS>
 
<ADDRESS>Copyright 2006  OC Systems, Inc.</ADDRESS>
 
<P></P>
 
<P></P>
  
</DIV>
 
 
 
 
 
 
<!-- footer -->
 
</div>
 
 
----
 
----
 
Copyright 2006-2017 OC Systems, Inc.</div>
 
Copyright 2006-2017 OC Systems, Inc.</div>

Revision as of 05:04, 15 September 2017


Next Previous Index Top

RootCause User Guide

RootCause Demo

-

RootCause Demo


This demonstration program, included as part of the RootCause installation, has been designed to provide an introduction and overview of the RootCause product. The program is:

$APROBE/demo/RootCause/C++/pi_demo

It is a simple 2-file C++

program, which computes the value of Pi by iteration using multiple threads. You can find the source in the same directory as the executable

file.

The goal of this demonstration is to provide an overview of the whole RootCause process, showing initial definition and tuning of the trace, then collection and viewing of more detailed data about a specific function. The demonstration pictured in this chapter was performed on Solaris. Output should be very similar on AIX and Linux.

Set Up

Before running this RootCause Demo, you must install and set up to use RootCause as described in [["rcc-7.html#MARKER-9-508">Chapter 4, "Getting Started". This will define the APROBE environment variable which is necessary to use RootCause.

In the instructions that follow, we'll use $APROBE to refer to the path where RootCause is installed, for example /opt/aprobe.

Use a Local Disk

We recommend you set your current directory to a disk local to the machine you're running on, though this is not required.

Defined X-Windows DISPLAY

Lastly, make sure your DISPLAY environment variable is set. If you're using a Windows client that is running X emulator software such as eXceed or Reflection, we recommend you move to a Unix display for your initial evaluation. If this is impractical, see [["rcc-11.html#MARKER-9-1878">"X-emulators: (Exceed, Reflection)".

Run With RootCause

Run the following commands:

rootcause_on
$APROBE/demo/RootCause/C++/pi_demo
<DIV ID=MARKER-10-714></DIV>rootcause_off

The rootcause_on command enables the automatic logging of every process that is started, and rootcause_off disables this logging. When the pi_demo program

is "[["rcc-6.html#MARKER-9-451">registered" with RootCause, it will be traced according to your specifications as well as simply being logged. The illustration below shows the set up and demo execution..

<IMG ALIGN=TOP SRC="sol_cpp_run1.gif">

NOTE: On AIX, the above process is slightly different than shown above. The invocation of the program must be done with the [["rcc-12.html#MARKER-9-2119">rootcause run command directly. For example, the above sequence is changed to:

rootcause run $APROBE/demo/RootCause/C++/pi_demo

See [["rcc-7.html#MARKER-9-532">"Enabling RootCause for an AIX Application".

View the RootCause Log

Enter the following command:

rootcause open

This will open the RootCause main window, and then a view of the RootCause log. This shows the [["rcc-11.html#MARKER-9-1725">Trace Display window, the window for viewing all trace events. On the left is the Event Tree; on the upper right is the source/text window; and on the lower right is the "details" window. In the text window you will see some information about the log file.

Locate the

APP_START event (in the Trace Event window) associated with the pi_demo program

run earlier. To view information about this event, select the APP_START node in the event tree with a left click

. This will fill in "details" about that event in the lower right window.With the APP_START pi_demo

node highlighted, right-click

to bring up the [["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu.

Click [["rcc-11.html#MARKER-9-1730">Open Associated Workspace in the popup-menu of the pi_demo

APP_START event.

.

<IMG ALIGN=TOP SRC="sol_cpp_log1.gif">

This will open a [["rcc-11.html#MARKER-9-1434">New Workspace Dialog with the program name and default

workspace filled in.

This combination: selecting a node in the tree, then using the popup menu to choose an operation, is the basic way of working within RootCause.

Create a RootCause Workspace

To complete the creation of a RootCause workspace for the

pi_demo application:

  1. Click Ok in the New Workspace dialog to complete the creation of the workspace.

<IMG ALIGN=TOP SRC="sol_cpp_new.gif">

  1. Click Yes to confirm that you want to

    register the pi_demo application
    with this workspace. This registration is how RootCause determines what applications to trace.

<IMG ALIGN=TOP SRC="sol_not_reg.gif">

You may close the RootCause Log window opened in a previous step.

You now see the

RootCause [["rcc-11.html#MARKER-9-1321">Workspace Browser. This is described in detail in [["rcc-11.html#MARKER-9-1320">Chapter 8, "RootCause GUI Reference".

<IMG ALIGN=TOP SRC="sol_cpp_wsp.gif">

Define the Trace

There are several aspects to a RootCause trace:

  • Predefined UALs selected from the [["rcc-11.html#MARKER-9-1323">Workspace Tree;

  • Method and Line traces selected from the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog;

  • Probes to gather or preserve data, also selected in the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog; and

  • User-written custom probes as described in [["rcc-13.html#MARKER-9-2198">"Writing Custom Probes".

In this part of the demo we illustrate the use of Predefined UALs and tracing function

calls; then we'll return to the Trace Setup Dialog to add some probes. Custom probes are an advanced feature not presented here.

In the RootCause GUI main window, opened in the previous section:

Enable a UAL

Under the [["rcc-11.html#MARKER-9-1330">UALs node in the [["rcc-11.html#MARKER-9-1323">Workspace Tree:

  1. Check the checkbox next to predefined UAL labeled
    exceptions.ual
    , as highlighed in the figure above. Simply checking this box will report all user-defined C++ (and Ada) exceptions.

Define Function
Traces

Now we'll add traces specific to this application.

  1. Click on the [["rcc-11.html#MARKER-9-1426">Setup button in the button bar.

This will open the

[["rcc-11.html#MARKER-9-1531">Trace Setup Dialog, showing the modules of the application in the

[["rcc-11.html#MARKER-9-1533">Program Contents Tree. The [["rcc-11.html#MARKER-9-1533">Program Contents Tree identifies the

modules,

files,

functions, methods, and

data

in the program, and allows you to specify complex actions on each function and method.

For this demo we'll first just specify a trace on the functions in the file pi_demo.cpp

, then return later to add data and probes.

  1. Click on the "lever" icon next to the pi_demo module to expand it; then expand the /tmp/demo directory, so you see the nodes for the source files pi.cpp and pi_demo.cpp.

  2. Click on the pi_demo.cpp source file node, then right-click to see the [["rcc-11.html#MARKER-9-1545">Trace Setup Popup Menu.

  3. Click [["rcc-11.html#MARKER-9-1557">Trace All In pi_demo.cpp

    .

  4. Click the OK button at the bottom right of the dialog to record the trace and dismiss the Trace Setup dialog.

<IMG ALIGN=TOP SRC="sol_cpp_setup1.gif">

Trace With RootCause

As was done in the preceding section,[["#MARKER-9-713">"Run With RootCause", run the following commands:

<DIV ID=MARKER-2-760></DIV>rootcause_on
$APROBE/demo/RootCause/C++/pi_demo
<DIV ID=MARKER-10-761></DIV>rootcause_off

Or, for

AIX:

<DIV ID=MARKER-2-763></DIV>rootcause run $APROBE/demo/RootCause/C++/pi_demo

This time, since the program is registered with a workspace, it will be traced as specified in the workspace, and the resulting output will be recorded within the workspace. There will be some startup delay, but if you notice that the program runs slower once started, this is probably because your workspace is being accessed across the network from your machine. See [["rcc-13.html#MARKER-9-2164">"RootCause and Efficiency Concerns".

View The Data Index

We're now ready to view the data generated by running with our Trace. This is discussed in detail in [["rcc-6.html#MARKER-9-366">"RootCause Data Management".

In the [["rcc-11.html#MARKER-9-1321">Workspace Browser window, do the following:

  1. Click the [["rcc-11.html#MARKER-9-1431">Index button. This will bring up the [["rcc-11.html#MARKER-9-1680">Trace Index Dialog for the most recently generated data.

  2. In the Trace Index Dialog, click the [["rcc-11.html#MARKER-9-1699">Select Events button.

  3. In the [["rcc-11.html#MARKER-9-1711">Select Events Dialog, check "Exceptions" to index exception events, then click [["rcc-11.html#MARKER-9-1718">Update to close the dialog and refresh the Index. You should now see something like the dialog below:

<IMG ALIGN=TOP SRC="sol_cpp_index1.gif">

  1. In the Trace Index Dialog, double-click on the second item in the table, the first item with an Event name of C++ EXCEPTION.

This will open a Trace Display Dialog centered at that

C++ Exception event, as shown below.

<IMG ALIGN=TOP SRC="sol_cpp_log2.gif">

Examine and Revise the Trace

The C++ Exception

trace event selected from the index should appear highlighted in the Trace Display. This was a result of checking

exceptions under [["#MARKER-9-741">"Enable a UAL" above.

The

ENTER and EXIT nodes are a result of the "Trace All In

pi_demo.exe" action added under [["#MARKER-9-744">"Define Function Traces" above.

You can see more information about the exception in the [["rcc-11.html#MARKER-9-1830">Event Details Pane in the lower right

.

If you scroll to the top of the Event Trace, you should see three threads: one for the main thread, and one for each of the threads which is started to compute Pi.

The

event tree is a

call tree, and can be very useful. From an

ENTER or

EXIT node in the tree you can use the [["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu to:

  • remove the called function

    from the set of function
    s to be traced,

  • find the next reference to the same function

    in the trace events, or

  • go to the function

    in the
    [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog
    to trace additional information.

As you step to each event, the [["rcc-11.html#MARKER-9-1830">Event Details Pane may show additional information about that event. See [["rcc-11.html#MARKER-9-1725">"Trace Display" for a more complete description of this window.

Take some time to explore the event tree. Then we will look at using the information available here to revise or "

tune" the trace used in the next run.

Call Counts

Useful information about the function

s called in your program may be obtained by looking at the call frequency as shown in the CALL_COUNTS table.

  1. Select (left-click) the S

    YN_
    CALL_COUNTS node, near the end of the event tree.

  2. Right-click (with MB3) to show the [["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu on this node.

  3. Click [["rcc-11.html#MARKER-9-1761">Show Associated Table. This will open a table listing each called function

    and the number of times it was called.

  4. Select (left-click) the extern:"_start()"

    entry in the call count table.

  5. Right click to show the popup menu.

  6. Select [["rcc-11.html#MARKER-9-1746">Deselect Function In Trace Setup in the popup menu.

You can also [["rcc-11.html#MARKER-9-1759">Find Function In Trace Events to search for functions

in the call tree, and remove them from there.

You can also search for functions

in the call tree, and remove them from there.

<IMG ALIGN=TOP SRC="sol_cpp_calls.gif">

  1. When you've finished making changes to the trace, click the Dismiss button at the bottom of the S

    YN_
    CALL_COUNTS table window, and then

  2. Click the

    [["rcc-11.html#MARKER-9-1427">Build button in the main window.

  3. Notice the effects of removing these calls in the next trace we generate.

NOTE: In most "real" programs, high-overhead functions selected for tracing are automatically identified and disabled via [["rcc-6.html#MARKER-9-433">load shedding, and are listed in the [["rcc-11.html#MARKER-9-1857">LOAD_SHED Table associated with the LOAD_SHED node at the end of the event tree. This demo doesn't run long enough for the load shedding heuristics to apply. See [["rcc-6.html#MARKER-9-374">"RootCause Overhead Management" for a general discussion of load shedding.

Tracing The Details

So far we have seen how the RootCause process works by:

  • enabling a predefined UAL (

    exceptions);

  • defining a simple trace from the Trace Setup Dialog;

  • running the application under RootCause;

  • choosing an event in the Trace Index Dialog;

  • viewing events in the Trace Display; and

  • modifying the trace by selecting functions

    from the call counts table.

RootCause allows you to record much more than the entry and exit of functions

and threads. You can record data values and insert probes as well.

Open Trace Setup

  1. Click the [["rcc-11.html#MARKER-9-1426">Setup button in the Workspace Browser to return to the [["rcc-11.html#MARKER-9-1531">Trace Setup Dialog.

    This is just as in [["#MARKER-9-738">"Define the Trace" above, but this time we'll record some details about a specific function

    ,
    ::ataninvint
    .

Select A Single Function

  1. Click the "lever" to expand the M pi_demo node to see the Source Files in the module.

  2. Expand the

    pi.cpp
    node to see the methods.

  3. Select the ataninvint method. This will bring up the source code for this method in the [["rcc-11.html#MARKER-9-1574">Source Pane, and show a tree of data to log in ataninvint

    in the [["rcc-11.html#MARKER-9-1579">Variables Pane.

Log Parameters

  1. Check the box for variable X in the Visible On Entry subtree, and for the return value in the Visible On Exit subtree.

<IMG ALIGN=TOP SRC="sol_cpp_vars.gif">

Add a Snapshot Probe

  1. In that same lower-right area, click on the

    Probes tab to show the [["rcc-11.html#MARKER-9-1589">Probes Pane. Probes in this context are special [["rcc-6.html#MARKER-9-396">actions that can be performed at points in the currently selected function
    .

  2. Click the "On" checkbox.

  3. Where it says No Trigger, select Function Entry.

  4. Where it says No Action, select

    Log Snapshot.

  5. Where it says ROOTCAUSE_SNAPSHOT, select this and type in "My Snapshot" and hit Enter.

<IMG ALIGN=TOP SRC="sol_probes.gif">

We've now requested that the parameter X

be logged (recorded) on entry to method

ataninvint, and also that a data snapshot be taken at this point and marked with the Event Name "My Snapshot".

Note: A snapshot causes data which might otherwise be deleted do to "data wraparound" to be preserved. In this small demo, a snapshot is not really necessary since it doesn't generate enough data to wrap around and cause old data to be lost. See [["rcc-6.html#MARKER-9-371">"Data Snapshots" for more information.

Save and Build the Trace

  1. Click the OK button at the bottom right of the dialog to save and build the trace and dismiss the Trace Setup dialog. Note that this will take a bit longer this time because we've created Probe [["rcc-6.html#MARKER-9-396">actions that require compilation of an [["rcc-6.html#MARKER-9-390">APC file.

Run With RootCause

  1. Again, check that rootcause is enabled with either the [["rcc-12.html#MARKER-9-2094">rootcause_on or [["rcc-12.html#MARKER-9-2129">rootcause status command. Then run the application by running the pi_demo program

    , as described in [["#MARKER-9-759">"Trace With RootCause".

Index the New Trace

  1. Click the [["rcc-11.html#MARKER-9-1431">Index button in the Workspace Browser window.

  2. In the Trace Index Dialog that comes up, double-click on the SNAP entry labeled "My Snapshot" to go right to where our new probes were added.

  3. Select the "ENTER ataninvint" node immediately preceding the Snapshot node to see the value of x

    on entry.

  4. Select the corresponding "EXIT ataninvint" node following the Snapshot to see the return value on exit.

Find the Calls Of Interest

The Trace Display window opened from the Trace Index Dialog operation will contain many events, including other calls to ataninvint.

  1. With the "ENTER ataninvint" node selected, right-click to show the [["rcc-11.html#MARKER-9-1826">Trace Display Popup Menu and choose [["rcc-11.html#MARKER-9-1759">Find Function In Trace Events.

  2. This will bring up the [["rcc-11.html#MARKER-9-1842">Find Text in Trace Events Dialog with the current function

    name filled in. Click Next, and the next occurrence of this string will be selected. Thereafter you can continue clicking Next, or enter any other string to search for. You can use [["rcc-11.html#MARKER-9-1763">Find Text in Trace Events from any the popup or Edit menu to search for any string in the current Trace Display.

<IMG ALIGN=TOP SRC="sol_cpp_log3.gif">

Where To From Here?

This chapter should have given you a good overview of the process of developing a trace and gathering data for a program. Now you're ready to try it on your own application. The application you use should be a "debug build" so Trace Setup has the information it needs. See [["rcc-13.html#MARKER-9-2163">"Selected Topics".


[[rcc-9.html>[Next]

[[rcc-7.html>[Previous] [[rcc-1.html>[Top] [[rcc-3.html>[Contents] [[rcc-14.html>[Index]

<ADDRESS>Copyright 2006 OC Systems, Inc.</ADDRESS>


Copyright 2006-2017 OC Systems, Inc.

Next Previous Index Top