750347 - [meta] Record JavaScript code coverage of automated tests

Status: NEW

(Testing :: Code Coverage, defect)

Description

[Gregory Szorc [:gps]]

It would be splendid if the build system measured and recorded code coverage of the in-tree JavaScript tests. This includes xpcshell and mochitests.

Some people have hacked this up before. I'd like to see it officially integrated with the build system as a build target (e.g. `make xpcshell-test-coverage`) and eventually have the results recorded by the TBPL build infrastructure.

This bug will be a meta/tracking bug. Technical discussion will likely occur in child bugs.

Comment 1

[Joshua Cranmer [:jcranmer]]

I have some results on running code coverage on comm-central's mailnews xpcshell tests here:
<https://www.tjhsst.edu/~jcranmer/test-ui/coverage.html?dir=mailnews> (selecting mailnews here because a lot of the mozilla code wasn't tested/run). I can already tell you by looking at that coverage that there appears to be one xpcshell test which is broken (!).

The necessary infrastructure is:
1. A tool to dump a map of runtime URLs to source file locations in the tree
2. A tool to post-process the output LCOV files to remap file locations as mentioned above
3. A tool to calculate the uncoverage of code not run.

I have prototypes of all of these pieces that I used to build the aforementioned site.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

there are 2 ways to record this:
1) emitting lcov output from the JSVM, there is some work here with a small list of known bugs remaining (linux64-ccov)
2) using devtools debugging api to get coverage, this works in general, but there are a small list of remaining issues (linux64-jsdcov)

using jsdcov we can capture per test info easier, but we lack some of the deeper collection of coverage.

I would like to know what we want to use this bug to track (or duplicate it to).  if this is JSVM coverage, we have a tracking bug for that (bug 1189360) and if this is jsdcov, we have a tracking bug for that (bug 1301174).

Comment 3

[Gregory Szorc [:gps]]

I have no clue. Maybe we should make bug 1189360 and bug 1301174 block this one, effectively turning this bug into a master tracker and those other bugs into trackers for the tech-specific coverage mechanism?