Brcov Predefined Probe

Branch Coverage Predefined Probe: brcov.ual

NOTE: This predefined probe is still under development and not all features and platforms are supported.

The brcov.ual predefined probe supports branch coverage analysis on user applications. Branch coverage tells you exactly which branches within a function were actually hit during program execution. This allows you to develop unit test cases which can be used to confirm that all the branches of code you have written are executed and taken.

While this probe is a unique and powerful tool for collecting branch coverage information, it is only a tool. Getting complete branch coverage on a function can be a difficult task, and requires the user to compose test cases which will cause all paths to be executed, including error conditions and exception propagation. The brcov.ual probe can be used with the coverage.ual to produce comprehensive test coverage information.

But it should be noted that it is in testing these tricky cases that Aprobe can especially useful, since you as a user can write a probe that modifies the data on_entry to a function to force a specific path through the function, without worrying about how to get the normal caller of that function to pass that erroneous data.

You can find an example that works through some of the issues that arise from using Aprobe to do this in the $APROBE/examples/learn/branch_coverage directory. Additionally you are encouraged to contact OC Systems to discuss using Aprobe on your specific application.

Usage

This probe is applied at run time using aprobe as described under Branch Coverage UAL Parameters below. The only functions for which data will be collected are functions selected in the configuration file (see Branch Coverage Configuration File). Selecting a function means that branches in the function will be instrumented. Conversely, branches in functions that are not selected will not be instrumented, and no record will be kept of invocations of these functions or branches.

The instrumentation of branches requires that there be accurate source line debugging information recorded in the executable. This generally means that the application or library containing the covered functions:

• must have been compiled with the appropriate debug flag (-g);
• should have been compiled without optimization; and
• must have the debug information still available.

A snapshot of the current state of collected branch coverage data may be written to the APD file while the program is running (see Configuration of Snapshots). Taking such periodic snapshots at known events, like the entry to or exit from a particular function, may help correlate state data with the execution path recorded by the brcov probe. A final snapshot occurs automatically at program termination.

After the application program terminates, the apformat command must be run to produce an XML branch coverage .brc file from the collected data. Use the abrmerge tool to produce reports and from the .brc files. Data from multiple runs may be merged into a single .brc file and a single report using Abrmerge (see abrcmerge below). Coverage data from brcov.ual and coverage.ual can be combined using Abrmerge to produce very extensive reports.

The report produced by the abrcmerge consists of several parts: the "Overall Summary," the "Subprogram Summary," the "Subprogram Details," and the "Source Annotation." (A subprogram is another name for a function, procedure, subroutine -- any callable entity.) The summaries lists the total lines and branches in the subprogram, how many lines/branches were not instrumented nor executed, and the percentage of instrumented lines/branches that were executed for the subprogram. The details section reports whether the subprogram was called, and if it was, it will report how many of the instrumented lines/branches were executed. In cases where coverage is less than 100% for a particular subprogram, the Details section will also list instrumented lines/branches which were not executed. The source section will annotate any provided source files with branch/line coverage information which will assist with test case creation. Some parts of the report are options. The coverage report will look similar to that in Branch Coverage Summary Report.

Branch Coverage UAL Parameters

brcov.ual is specified on the aprobe command line or in an APO file, and with apformat. The specific options are:

aprobe   -u brcov[.ual]    [-p " [-c config_filename] [-h] [-v]"   your_program 

For example:

aprobe -u brcov -p "-c foo1.brcov.cfg" foo.exe
aprobe -u brcov -p 

where:

-c config_filename

specifies that the name of the probe configuration options file will follow immediately after -c. The default file name is your_program.brcov.cfg. For example, if your executable program is called wilbur.exe, then the default file name would be wilbur.exe.brcov.cfg.

-h

produces brief help text.

-v

verbose mode, which produces additional progress messages.

Note that if you use no UAL parameters, you need not specify UAL names at all. For example:

apformat progname

will format progname.apd with all the UALs with which it was created.

Branch Coverage Configuration File

The Branch Coverage configuration file is used to specify what subprograms are to be analyzed, when snapshots are to be taken, and other options, as described in Configuration File. The example below shows one possible Branch Coverage configuration file.

PROBE CONFIGURATION FILE FOR BRCOV VERSION 1.0.0

CoverageEnabledInitially TRUE

// Here we select which subprograms we want to cover:
COVERAGE "main"
COVERAGE extern:"YET_ANOTHER_LOCAL_PROC"
COVERAGE "STILL_ANOTHER_LOCAL_PROC"

// Here we define which subprogram invocations cause snapshots
// to be saved.
SNAPSHOT "printf" IN "libc.a(shr.o)" ON ENTRY IS "printf called"

Example D-2. brcov.cfg File

Note that if you do not provide a configuration file, the probe will create a default configuration file for you. This is a good way to get started.

Configuration Variables

The following are the only valid keywords that identify lines to set configuration variables. Each such line must begin with one of these keywords, and the keyword must be followed by a value. Nothing else is allowed on the same line.

CoverageEnabledInitially

This must be followed by the value TRUE or FALSE. The default is TRUE which indicates that data logging will begin as soon as the application program starts running. If set to FALSE, data logging will begin only after a call is made to the probe's function ap_Brcov_Enable(), rather than as soon as the application program starts running.

ShowAllSnapshots

This must be followed by TRUE or FALSE. The default is FALSE. When FALSE, the data from all intermediate snapshots are rolled into the one final report. When TRUE the data for every snapshot will be reported in separate summaries.

ResetSnapshotCounts

This must be followed by TRUE or FALSE. The default is FALSE. When FALSE, the data from all intermediate snapshots accumulate into the one final report. When TRUE the counts are reset for each separate snapshot.

ShowUnconditionalBranches

This must be followed by TRUE or FALSE. The default is FALSE, which indicates that unconditional branches will not be tracked. In general, conditional branches are more interesting than unconditional branches.

Configuration of Branch Coverage Functions

Each function for which branch coverage data is to be collected must be specified explicitly using the COVERAGE, COVERFILE, or COVERUNIT keywords.

The COVERAGE keyword is followed by the name of the function, as described in Configuration of Selected Functions. This directive selects specific functions by name (with possible wildcards).

The COVERFILE keyword is followed by the name of a file and an option module name. This keyword selects the functions originating from the designated source file (with possible wildcards).

The COVERUNIT keyword is followed by the name of a PowerAda unit. This keyword selects the functions originating from the designated PowerAda unit (with possible wildcards).

By default, if no COVERAGE/COVERFILE/COVERUNIT lines exist, then coverage data will only be collected for the "main()" function of the application module.

The REMOVE keyword allows you to specify subprograms that should not be instrumented for logging coverage data. This is useful when used in conjunction with a wildcard ("*"), to gather data about everything except certain routines.

Note: COVERAGE "*" is not a particuarly useful concept since it is important for you to identify and isolate your functions carefully in order to use test coverage for its intended purpose.

Configuration of Snapshots

The branch coverage probe configuration file allows you to specify the name of some subprograms for which snapshots of the coverage data are to be automatically taken. This is done with lines beginning with the keyword SNAPSHOT.

Each SNAPSHOT line must specify a particular subprogram in the usual manner, just as is done for a COVERAGE line.

The remainder of the SNAPSHOT line contains pairs, where each pair has a special identifier keyword followed by its own associated value. These pairs give supplementary information about the snapshot.

• ON - This optional special identifier must be followed by the value ENTRY or EXIT, to denote that the snapshot is to be taken on entry to or exit from the subprogram respectively.

• IS - This optional special identifier must be followed by an (arbitrarily long) string enclosed within quotation marks (""). It specifies a textual description or title that is to be logged along with the snapshot.

Test Coverage API

Users can control the behavior of the coverage probe by calls from within their own probes. The API for the trace probe is defined by [../include/coverage.h $APROBE/include/brcov.h]. Some of the functions exported by brcov.ual are:

• ap_Brcov_Enable - Enables logging of coverage data.
• ap_Brcov_Disable - Disables logging of coverage data.
• ap_Brcov_DoSnapshot - Dumps current coverage information.

See Snapshots for more information and an example.

Branch Coverage Performance Issues

See Performance Issues for a general discussion of factors that affect performance.

The branch coverage probe is applied to the functions specified in the configuration file, so the run-time overhead is directly proportional to the number of calls to the functions selected and the number of branches executed in each function. The branch coverage probe simply increments an integer for each branch executed, so the amount of data is a constant multiple of the number of branches being covered, though additional snapshots increase this.

As branch coverage analysis is generally a unit-testing activity, it is mostly done on the small number of related functions under test, so performance should not be an issue. Application-wide branch coverage analysis during integration testing is not recommended.

Abrmerge

The results of multiple runs of brcov.ual (and coverage.ual) over the same executable may be merged into a single "combined summary" report using abrmerge.

A merged summary report is useful when running many tests, each with different input data or perhaps different probes, to force all logic paths to be executed. abrmerge can also merge branch data files to accumulate data from multiple runs into a single file.

A simple example

NOTE: this does not exist yet.

This example of the coverage predefined probe can be found in $APROBE/examples/predefined_probes/brcov.

The test application is also very simple. It just prints out a mini-menu and waits for you to select 1 or 2. Depending on your selection, it does one of two actions. You can see this in the DoProcessing() function in coverage_example.c. We want to exercise every branch in this function -- we refer to this as getting 100% coverage.

Note that, unlike some other uses of Aprobe, branch coverage requires you to be familiar with (the lines in) your source code to make most effective use of the probe.

As before, we'll use make to build the example application and then we'll create a configuration file to select the function:

cd $APROBE/examples/predefined_probes/brcov
make
cat > brcov_example.brcov.cfg
PROBE CONFIGURATION FILE FOR BRCOV VERSION 1.0.0
COVERAGE "DoProcessing()"
COVERAGE "main()"
^D
aprobe -u brcov -p brcov_example

When the example application prompts you for a choice, enter the number 1. That will print out the first string and then the application will exit normally.

As before, aprobe stored the data in .apd files, so use apformat to view it:

apformat brcov_example

Here's a portion of the formatted output showing the branch coverage we've achieved:

TBD

Using abrmerge

Sometimes we want to combine several test and branch coverage reports into a single report. Use abrmerge to accomplish this.

The above 2 lines that were not executed are specific to choice number 2. We could easily execute the lines of choice number 2 if we ran it again, but then we wouldn't be able to cover the choice 1 lines. If you look in the local directory, you'll see a coverage_example.tc file. This was created at format time by the coverage predefined probe and is a binary representation of the above data. It contains the coverage data obtained during choice 1, so let's save it.

We typically use abrmerge to merge results from many .brc files (perhaps .brc files produced by coverage.ual and brcov.ual) and display the combined data, but it can also act on just one file. We need to save the current .brc file, and copying it would do, but we'll use abrmerge to demonstrate it a little:

abrmerge -b combined.brc brcov_example.brc

This merges the data in brcov_example.brc with that in combined.brc (which, in this case, means that combined.brc ends up with the same data). You can verify that this combined.brc file has the same data by just running abrmerge on it:

atcmerge combined.brc

Now run the application again using aprobe:

aprobe -u brcov brcov_example

Select choice 2, then after the application ends re-run apformat:

apformat brcov_example

This time we get a different set of lines executed, as expected. Use the same abrmerge command to combine the data in combined.brc (from the first run) with the data from this run (in a new brcov_example.brc):

abrmerge -b combined.brc brcov_example.brc

Now we get the 100% coverage we wanted!

The on-line example in $APROBE/examples/learn/branch_coverage extends this by illustrating how to use your own probes in combination with coverage.ual to ensure all lines are executed.

Branch Coverage Report

The final output of the learn/branch_coverage' example looks like this:

Coverage Summary Report

TBD