Coverage data merging

From OC Systems Wiki!
Revision as of 20:28, 29 March 2017 by Swn (talk | contribs) (Swn moved page Coverage merging to Coverage data merging: better name)
Jump to: navigation, search

Coverage Data Merging

If you need to merge coverage data from several test runs with the same or different test applications/drivers you can use the atcmerge and/or apbc tools.

The coverage UAL produces two types of output:

  • .tc - test coverage file
  • .bc - detail coverage file

The files are produced by running apformat on the .apd file which is produced by running aprobe. For example:

aprobe -u coverage test1.exe
apformat test1.apd

Note that if you run the same test application/driver multiple times, you need to save the .tc and .bc files between each apformat operation:

aprobe -u coverage test1.exe
apformat test1.apd
mv test1.tc test1-run1.tc
mv test1.exe.bc test1.exe-run1.bc
aprobe -u coverage test1.exe
apformat test1.apd
mv test1.tc test1-run12tc
mv test1.exe.bc test1.exe-run2.bc

You can merge the .tc coverage data with the following command:

atcmerge -b all.tc test1.tc test2.tc test3.tc

This will merge all the .tc files into a new file all.tc. Atcmerge will also generate a coverage summary report of the merged coverage data to stdout.

You don't need to merge .bc data before displaying. You can use the following command to display merged .bc coverage data:

apbc test1.exe.bc test2.exe.bc test3.exe.bc file1.adb file2.adb file3.adb

This will merge the detailed coverage data from the three .bc files and display line counts associated with the source lines of the three Ada source files.

In Aprobe 4.4.6d you can produce a merged .bc file using atcmerge at the same time you merge the .tc files:

atcmerge -b all.tc -d all.bc test1.tc test2.tc test3.tc

This will merge all the .tc files into all.tc and produce a new .bc file all.bc wth the merged data.

Example

The following is an example of merging coverage data. In this example the code under test in the Ada package named pkg. Three test applications (test1.exe test2.exe, and test3.exe) were created to exercise this code, each one performing a set of tests. The coverage data from running all three test applications will then be merged to produce a single summary report and source report.

First, we make three test runs. (We don't have to worry about .tc or .bc files being overwritten because the test applications have unique names.)

aprobe -u coverage test1.exe 
apformat test1.apd
aprobe -u coverage test2.exe
apformat test2.apd
aprobe -u coverage test3.exe
apformat test3.apd

This will have produce six coverage data files:

  • test1.tc and test1.exe.bc
  • test2.tc and test2.exe.bc
  • test3.tc and test3.exe.bc

We can merge the coverage data and produce a overall summary report with:

atcmerge -b all.tc test1.tc test2.tc test3.tc > all.rpt

We can merge the detail coverage data and produce a source report with:

apbc test1.exe.bc test2.exe.bc test3.exe.bc pkg.ads pkg.adb 

Here are the reports:

Test Coverage Summary Report


PID 790542 - Test Run Summary:

Execution Start Time      End Time                  Original Report File
------------------------  ------------------------  ------------------------
Wed Mar 29 15:52:41 2017  Wed Mar 29 15:52:41 2017  main1.tc
Wed Mar 29 15:52:42 2017  Wed Mar 29 15:52:42 2017  main2.tc
Wed Mar 29 15:52:42 2017  Wed Mar 29 15:52:42 2017  main3.tc

End of Test Run Summary.


List of application modules for this report:

main1.exe
main2.exe
main3.exe


Subprogram Summary: Merged Coverage Data

Module <multiple application modules>:
-------------------------------------
                                                             Lines   Lines
                                             Total   Total   Not     Not      Coverage
Subprogram Name                              Calls   Lines   Probed  Executed Percent 
--------------------------------------------------------------------------------------
extern:"pkg.td.body_elab" in main3.exe           3       0        0         0        -
extern:"pkg.td[1]" in main3.exe                  4      11        0         1       91
extern:"pkg.body_elab" in main3.exe              3       5        0         0      100
extern:"pkg.s1[1b]" in main3.exe                 2       1        0         0      100
extern:"pkg.s2[1b]" in main3.exe                 2       1        0         0      100
extern:"pkg.put[1b]" in main3.exe                2       2        0         0      100
extern:"pkg.get[1b]" in main3.exe                2       2        0         0      100
extern:"pkg.p1[1]" in main3.exe                  2       6        0         0      100
extern:"pkg.p2[1]" in main3.exe                  2       6        0         0      100
extern:"pkg.spec_elab" in main3.exe              3       4        0         0      100
extern:"pkg.s1[1]" in main3.exe                  2       1        0         0      100
extern:"pkg.s2[1]" in main3.exe                  2       1        0         0      100
extern:"pkg.put[1]" in main3.exe                 2       2        0         0      100
extern:"pkg.get[1]" in main3.exe                 2       2        0         0      100

--------------------------------------------------------------------------------------
Total                                           33      44        0         1       98


End of Subprogram Summary.


Subprogram Details: Merged Coverage Data


Module <multiple application modules>:
-------------------------------------
For 3 calls to subprogram: extern:"pkg.td.body_elab" in main3.exe
  Lines in subprogram: 0

For 4 calls to subprogram: extern:"pkg.td[1]" in main3.exe
  Lines in subprogram: 11 total (incl. executed 10 of 11 probed lines)
  Percent of lines executed:  91
  The following probed lines were not executed:
    Line: 20 (/home/swn/aprobe/test/faq/covada/pkg-td.adb)

For 3 calls to subprogram: extern:"pkg.body_elab" in main3.exe
  Lines in subprogram: 5 total (incl. executed 5 of 5 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.s1[1b]" in main3.exe
  Lines in subprogram: 1 total (incl. executed 1 of 1 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.s2[1b]" in main3.exe
  Lines in subprogram: 1 total (incl. executed 1 of 1 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.put[1b]" in main3.exe
  Lines in subprogram: 2 total (incl. executed 2 of 2 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.get[1b]" in main3.exe
  Lines in subprogram: 2 total (incl. executed 2 of 2 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.p1[1]" in main3.exe
  Lines in subprogram: 6 total (incl. executed 6 of 6 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.p2[1]" in main3.exe
  Lines in subprogram: 6 total (incl. executed 6 of 6 probed lines)
  Percent of lines executed:  100

For 3 calls to subprogram: extern:"pkg.spec_elab" in main3.exe
  Lines in subprogram: 4 total (incl. executed 4 of 4 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.s1[1]" in main3.exe
  Lines in subprogram: 1 total (incl. executed 1 of 1 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.s2[1]" in main3.exe
  Lines in subprogram: 1 total (incl. executed 1 of 1 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.put[1]" in main3.exe
  Lines in subprogram: 2 total (incl. executed 2 of 2 probed lines)
  Percent of lines executed:  100

For 2 calls to subprogram: extern:"pkg.get[1]" in main3.exe
  Lines in subprogram: 2 total (incl. executed 2 of 2 probed lines)
  Percent of lines executed:  100


End of Subprogram Details.

pkg-td.adb:
Hit Line  Source
--- ----  ------ - -
    0001: 
    0002: separate(pkg)
    0003: 
    0004: procedure td(i : in integer) is
  4 0005:   i0 : integer := 0;
  4 0006:   i1 : integer := 1;
  4 0007:   f0 : float := 0.0;
  4 0008:   f1 : float := 1.0;
    0009: begin
  4 0010:   case i is
    0011:   when 1 =>
  1 0012:     p1(i0);
    0013:   when 2 =>
  1 0014:     p1(i1);
    0015:   when 3 =>
  1 0016:     p2(f0);
    0017:   when 4 =>
  1 0018:     p2(f1);
    0019:   when others =>
  0 0020:     Text_Io.Put_Line("bad test number");
    0021:   end case;
  4 0022: end td;
    0023:   

pkg.adb:
Hit Line  Source
--- ----  ------ - -
    0001: 
    0002: with Text_Io;
    0003: 
    0004: package body pkg is
    0005: 
 15 0006:   package g2 is new gen(float);
    0007: 
  3 0008:   procedure p1(I : in out integer) is
    0009:   begin
  2 0010:     g1.put(i);
  2 0011:     if I = 0 then
  1 0012:       Text_Io.Put_Line("p1, 0!");
    0013:     else
  1 0014:       Text_Io.Put_Line("p1, not zero");
    0015:     end if;
  2 0016:     I := g1.get + 1;
  2 0017:   end p1;
    0018: 
  3 0019:   procedure p2(I : in out float) is
    0020:   begin
  2 0021:     g2.put(i);
  2 0022:     if I < 0.000001 and then I > -0.000001 then
  1 0023:       Text_Io.Put_Line("p2, 0!");
    0024:     else
  1 0025:       Text_Io.Put_Line("p2, not zero");
    0026:     end if;
  2 0027:     I := g2.get + 1.0;
  2 0028:   end p2;
    0029: 
  3 0030:   procedure td(i : in integer) is separate;
    0031: 
  3 0032: end pkg;
    0033: 

pkg.ads:
Hit Line  Source
--- ----  ------ - -
    0001: 
    0002: with gen;
    0003: 
    0004: package pkg is
    0005: 
 15 0006:   package g1 is new gen(integer);
    0007: 
  3 0008:   procedure p1(I : in out integer);
  3 0009:   procedure p2(I : in out float);
    0010: 
  3 0011:   procedure td(I : in integer);
    0012: 
    0013: end pkg;
    0014: