1281967 - NSS support for coverage testing

Status: RESOLVED FIXED

(NSS :: Test, defect)

Description

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Now that we have NSS builds and tests running on TaskCluster, it should be relatively easy to get continuous coverage monitoring, at least on Linux.  The “minimum viable product” I have in mind is browsable marked-up source for each push; analysis across versions would be a followup.  (It doesn't necessarily need to be every push, but with TaskCluster/TreeHerder in place it'd be more work to make it *not* every push.)

Things I'll need to do:

1. Allow building with --coverage.  NSPR shouldn't need it, and either SQLite should be excluded or I'll need to delete its .gcda file later on, because it's huge and not our problem.

2. Add a build type which enables that to automation/taskcluster/graph/linux.  (Varying word size / assertions / compiler / etc. doesn't seem too useful here, so only one build type.)

3. Export the *.gcno files from the build task as an artifact (if coverage) and retrieve/unroll them in the test tasks.

4. Add lcov to the packages installed in the VM.

5. In both build and test tasks (because the build runs the cert tests, as well as some tools like shlibsign), do the lcov --capture step and export the *.info file as an artifact.

6. Finally, add a task depending on all the build/test tasks which: downloads the coverage info artifacts, checks out the source, runs genhtml, and exports that as some kind of publicly visible artifact.  (Optional: add genhtml runs to the individual test jobs.  Not sure if there's much benefit vs. the increased storage use.)  It would be nice if the file tree could be directly browsable, but a single *.tar.bz2 or *.zip would still be more or less usable.

Comment 1

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: bug1281967-nss-gcov-ci-wip2016070100.diff

Work in progress, in git-format-patch format (base 6f938bc), complete with vague commit messages and FIXMEs (and a few things that should probably be deleted that let me work from Git instead of Hg and pull things from bind-mounted file:/// instead of remotely).  I tested the scripts by running docker containers locally by hand, but not the TC graph integration.  And that definitely won't work yet — the nss-ci image needs to be rebuilt with the "lcov" package added.

More importantly, I'm told that there isn't enough interest in actually using the output of this to justify enabling it; it could maybe still be landed disabled by default so that Try jobs could use it, but that feature would need to be implemented first.

(Possible future/related work: the same idea should work for Firefox's CI, and would have the side-effect of showing us what parts of NSS aren't being tested there.)

Comment 2

[Tim Taubert [:ttaubert] (inactive)]

Let's use sancov and start by making it easy to generate a coverage report on your local machine.

Comment 3

[Phabricator Automation]

Attachment: Bug 1281967 - Add "mach coverage ssl_gtests" using sancov r=franziskus

Comment 4

[Phabricator Automation]

Comment on attachment 8966907 [details]
Bug 1281967 - Add "mach coverage ssl_gtests" using sancov r=franziskus

Franziskus Kiefer [:fkiefer or :franziskus] has approved the revision.

https://phabricator.services.mozilla.com/D906

Comment 5

[Tim Taubert [:ttaubert] (inactive)]

https://hg.mozilla.org/projects/nss/rev/b8804e4c7989