Line coverage report using gcov/lcov

When using unit-tests to add robustness to development projects, it is nice (even important) to be able to see which parts of your code is actually being tested.

If you are using gcc, this is exceedingly easy to accomplish, using the built in tool gcov. All you need to do is use the compile flags -fprofile-arcs -ftest-coverage and link flags -lgcov -fprofile-arcs when compiling with gcc.

This creates a .gcno file for each compilation unit gcc processes. These files don't contain any actual coverage information (which makes sense, since no code has yet been executed). They only keep some basic data on code blocks and source code line numbers.

The key is that when a program built with gcov is executed, it creates .gcda files, one for each compilation unit. The .gcda-files contain the interesting coverage data -- which lines were touched, how many times, and which branches it went through. With the help of lcov, these files are processed to generate an html-page.

This scribble is divided into three parts:
1. A general outline of the steps to create a clean coverage report, which should apply to any gcc project.
2. How I arrived at these steps (cheating a bit for brevity)
3. Finally I'll summarize the working script, an implementation of the general outline.

1. General outline

  1. Clean project completely (you want a complete rebuild)
  2. Recompile everything with gcov compile and link flags.
  3. Use lcov to process initial .gcno files.
  4. Run all tests.
  5. Use lcov to process .gcna coverage files.
  6. Merge the initial and final lcov files (step 3 and step 5).
  7. Filter lcov output to only include project files.
  8. Filter lcov output to exclude unit test files, and main.cpp.
  9. Generate html coverage report from final filtered lcov output..

The point of step 3 and step 6 is to include files that have no coverage at all, and don't get a corresponding .gcna file. Without this, these untouched files wouldn't show up in the final coverage report.

2. Trial and error

First, a quick note on my SCons setup: All source files are contained in the ./src/ directory. During the build process, all used source code files are copied to a ./build/ directory, and all generated files (.o, .gcno, etc) also end up here.

This completely separates the build and src directories, keeping a clean shop, just how I like it.

I should also mention that my SCons script looks for environment variables CPPFLAGS and CCFLAGS, and uses these when present. So that step 1 and step 2 from the general outline become:

?
rm -rf ./build
export LDFLAGS="-lgcov -fprofile-arcs"
export CPPFLAGS="-fprofile-arcs -ftest-coverage"
nice scons -j6 --tests  # --tests is a custom option that builds tests
2.1 First attempt

Let's skip step 3 for now, and run the tests (step 4), process the coverage output (step 5), and generate the html report (step 9):

?
# Run tests
for i in ./bin/tests/* ; do $i ; done
# Process coverage files
lcov -c -d ./build -o coverage.run
# Generate html report (make sure ./html folder exists)
genhtml -o ./html/ coverage.run

The result is quite messy. For some reason, it includes coverage on boost, cxxtest and gcc-stl source files.

2.2 Second attempt - Excluding external files
 Let's clean this up with some filtering that only includes files in the project (step 7).
?
# Extract (-e) from coverage.run data from files in cwd
lcov -e coverage.run "`pwd`/*" -o coverage.run.filtered
genhtml -o ./html/ coverage.run.filtered

Much better. It still incorrectly lists directories that don't exist. Also, when browsing to the individual source files, it only shows the error message(missing file).

2.3 Third attempt - Specifying source directory

Turns out that lcov needs to be told what the base directory is, using the -b flag.

?
# Process coverage files
lcov -b . -c -d ./build -o coverage.run
# Extract (-e) from coverage.run data from files in cwd
lcov -e coverage.run "`pwd`/*" -o coverage.run.filtered
# Generate html report (make sure ./html folder exists)
genhtml -o ./html/ coverage.run.filtered

Almost there. Unit test files are still included in the coverage report. Since these will always have 100% coverage, it's better to not include them.

2.4 Fourth attempt - Excluding unit test files

Above we extracted our project files, now let's further remove all Test*.* files, which belong to the unit tests (step 8 ).

?
# Remove (-r) from coverage.run.filtered all Test*.* files
lcov -r coverage.run.filtered "`pwd`/*/Test*.*" -o coverage.run.filtered
genhtml -o ./html/ coverage.run.filtered

Only relevant data now. However, I know that there are several files that are not tested at all, and don't show up. Even a whole directory (audio) is missing.

2.5 Fifth attempt - Including static .gcno files

This is where step 3 and step 6 save the day. By using the-i flag it looks for the initial static .gcno files. If you remember, these were created when compiling, and there is one for each compilation unit. We merge this output with the one from running the tests, and filter the result like before.

?
# Process *.gcno files
lcov -b . -c -i -d ./build -o coverage.init
 
# Run tests like before
for i in ./bin/tests/* ; do $i ; done
 
# Process coverage from executed tests
lcov -b . -c -d ./build -o coverage.run
 
# Merge coverage.init and coverage.run
lcov -a coverage.init -a coverage.run -o coverage.total
 
# Filter like before, and also remove main.cpp
lcov -e coverage.total "`pwd`/*" -o coverage.total.filtered
lcov -r coverage.total.filtered "`pwd`/*/Test*.*" -o coverage.total.filtered
lcov -r coverage.total.filtered "`pwd`/build/main.cpp" -o coverage.total.filtered
genhtml -o ./html/ coverage.total.filtered

Finally, it shows my hitherto sloppy test coverage in all earnestness.

2.6 Sixth (final) attempt - Merging folders

The /build/ folder is just a by-product of compilation, a temporary mirror of files in the /src/ directory. It would be nice to pretend that the coverage output for the files in/build/ were located in /src/, merging together with the ones already there. How can that be done? I'm glad you asked!

?
sed 's/\/build\//\/src\//g' coverage.total.filtered > coverage.total.final
genhtml -o ./html/ coverage.total.final

That's right, a simple search-and-replace does the trick!

3. Summary - final script

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#!/bin/bash
 
# Step 1: Clean all build files
rm -rf ./build
 
# Step 2: Re-compile whole project, including tests
export LDFLAGS="-lgcov -fprofile-arcs"
export CPPFLAGS="-fprofile-arcs -ftest-coverage"
nice scons -j6 --tests
 
# Step 3: Generate initial coverage information
lcov -b . -c -i -d ./build -o .coverage.wtest.base
 
# Step 4: Run all tests:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`pwd`/lib
for i in ./bin/tests/* ; do $i ; done
 
# Step 5: Generate coverage based on executed tests
lcov -b . -c -d ./build -o .coverage.wtest.run
 
# Step 6: Merge coverage tracefiles
lcov -a .coverage.wtest.base -a .coverage.wtest.run  -o .coverage.total
 
# Step 7: Filtering, extracting project files
lcov -e .coverage.total "`pwd`/*" -o .coverage.total.filtered
 
# Step 8: Filtering, removing test-files and main.cpp
lcov -r .coverage.total.filtered "`pwd`/build/main.cpp" -o .coverage.total.filtered
lcov -r .coverage.total.filtered "`pwd`/*/Test*.*" -o .coverage.total.filtered
 
# Extra:  Replace /build/ with /src/ to unify directories
sed 's/\/build\//\/src\//g' .coverage.total.filtered > .coverage.total
 
# Extra: Clear up previous data, create html folder
if [[ -d ./html/ ]] ; then
    rm -rf ./html/*
else
    mkdir html
fi
 
# Step 9: Generate webpage
genhtml -o ./html/ .coverage.total
 
# Extra: Preserve coverage file in coveragehistory folder
[[ -d ./coveragehistory/ ]] || mkdir coveragehistory
cp .coverage.total ./coveragehistory/`date +'%Y.%m.%d-coverage'`
 
# Cleanup
rm .coverage.*

Note: As of lcov version 1.10, you can use --no-external in step 5and thus skip step 7.