Test Coverage Predefined Probe: coverage.ual Introduction ============ In this example we'll use of the Test Coverage predefined probe, coverage.ual. This probe, in combination with Aprobe's ability to alter a program's behavior, provides a uniquely powerful coverage-testing capability. For information about other predefined probes, see $APROBE/examples/evaluate/4.predefined. About Test Coverage =================== The idea behind "Test Coverage" is to prove that our test cases cover all lines within the code being tested. For example, a function may execute one block of code if its input parameter is null, and another when it's not null. But we want to be sure that each line of code is executed by at least one test case. If all our test cases (combined) cause execution of all the lines of code, we can say we have 100% test coverage. To determine when we have 100% test coverage, we need a tool to tell which lines have (or have not) been executed by a given test run. The coverage.ual predefined probe does this. Furthermore, we need a tool which summarizes which lines have or have not been tested during all the testing. This is done by Aprobe's atcmerge. But Aprobe goes one step further: it allows us to modify to whatever values we want, making it easy to force execution through paths that may be difficult to cover otherwise. And it does so without any need to change or recompile the program's source code. See Appendix D, Predefined Probe and Libraries for a complete description of coverage.ual and other predefined probes. Target Program ============== For this example (and several others), we'll use a single C++ class, OutFile which writes text to the screen or a file. This is defined in OutFile.h and implemented in OutFile.cpp. The program being tested is in TestOutFile.cpp. It creates an OutFile object called oFile using the parameterless form of the constructor, which writes to standard output. It is the program's main() function that we're trying to cover with our testing in this example. In particular, we want to get the lines that are executed when it runs with no errors, as well as the lines that are executed only as part of catching an exception when a Write operation fails. Note that the program must be compiled with -g so that line number information is available to Aprobe. Writing the Probe ================= We are going to use 2 different probes from 2 different UALs, at once. One will be coverage.dll, the other will not be a predefined probe. That's because our test program (see TestOutFile.cpp) will not execute all the lines in main() if it runs successfully, since the remaining lines are inside an exception handler and won't get executed unless main() catches an exception. So we're going to use a second probe, WriteException.apc, to force an exception to be raised so that main() can catch it. This probe works by intentionally passing an invalid input parameter to the OutFile::Write method, thus causing it to raise an exception. Since coverage.ual is predefined, we don't have to write any probes to use it. But we'll need to tell it the names of all functions whose test coverage we want to record. This is done via a "configuration file", the name of which is (by default) derived from the program being probed. In this case, it is TestOutFile.coverage.cfg, (at least these) and contains 2 lines: PROBE CONFIGURATION FILE FOR COVERAGE VERSION 2.0.0 coverage "main()" The first line the file identifies it as a coverage configuration file, and indicates that the only function we want to cover is "main()" in the application module itself. If you have Java installed on your machine, you can use the "Configuration GUI" to create or modify the configuration file. Again, see Appendix D: Predefined Probes and Libraries for more information. Running The Example =================== Since we're doing multiple test runs using multiple probes, the process is a bit different from most of the other examples. We'll use the Aprobe command line to perform the following steps: 1. Create the coverage configuration file as shown above, if necessary. 2. Run the program with coverage.ual. This gets coverage data for the program while no exception was raised. 3. Run apformat to see the coverage results for this run, and to create the binary test coverage result file, TestOutFile.tc. 4. Use atcmerge to merge the results of this run into the summary.tc file. This file will record the summary for all runs (you can use any filename with a .tc suffix). 5. Run apc to compile the WriteException.apc probe into WriteException.ual in preparation for the second run. 6. Run the test program again with aprobe using both coverage.ual and WriteException.ual. This gets coverage data for the program while an exception was raised. 7. Repeat steps 3 and 4 above (i.e., run apformat and atcmerge) so that summary.tc now contains the merged results of both runs. 8. See the merged results, which are also shown below. Note: These steps can be automatically performed for you by the Makefile for this example. Just type 'make test' at the command prompt to perform all the steps above while in this examples/learn/test_coverage directory to perform all the steps above. After the final atcmerge, the merged results look like this: Test Coverage Summary Report Test Run Summary: Execution Start Time End Time Original Report File ------------------------ ------------------------ ------------------------ Wed Apr 26 17:20:32 2000 Wed Apr 26 17:20:32 2000 Wed Apr 26 17:20:35 2000 Wed Apr 26 17:20:35 2000 End of Test Run Summary. Subprogram Summary: Module TestOutFile: ------------------ Lines Lines Total Total Not Not Coverage Subprogram Name Calls Lines Probed Executed Percent --------------------------------------------------------------- extern:"main()" 2 11 0 0 100 --------------------------------------------------------------- Total 2 11 0 0 100 End of Subprogram Summary. Subprogram Details: Module TestOutFile: ------------------ For 2 calls to subprogram: extern:"main()" Lines in subprogram: 11 total (incl. executed 11 of 11 probed lines) Percent of lines executed: 100 End of Subprogram Details. Summary ======= From this example you have learned: * What we mean by "Test Coverage" * How to use coverage.ual and atcmerge * How to use other probes in combination with coverage.ual to force your program to cover all paths