569228 - Record test coverage

Status: RESOLVED WONTFIX

(Firefox :: Sync, defect)

Comment 1

[Philipp von Weitershausen [:philikon]]

Attachment: Code coverage tool v1

Here's a first stab at a coverage tool for xpcshell tests. It consists of two parts:

1) A patch to the xpcshell harness that invokes the jsdIDebuggerService's interruptHook to get expression coverage. It records the data to a JSON file which is also used to transfer coverage state between individual tests (since they're independent processes). Coverage is turned on by providing the COVERAGE_OUTPUT environment variable (its value will be used as the filename for the JSON output file). One may also specify COVERAGE_FILTER which may be used to isolate coverage to only certain files. So far only resource:// URIs have been tested for this.

2) A simple analytics tool written in Python that reads the JSON file (requires either Python 2.6 or the simplejson library) and spits out a bunch of HTML files. Covered lines are green, uncovered ones red and ignored lines (comments, etc.) white.

Comment 2

[Philipp von Weitershausen [:philikon]]

Attachment: Example coverage report

Here's what the coverage tool spits out when run against my sandbox (not all of those tests have landed in the repo yet).

Comment 3

[Philipp von Weitershausen [:philikon]]

Attachment: Code coverage tool v1.1

Make work again after code reorg (bug 569355).

Comment 4

[Philipp von Weitershausen [:philikon]]

Attachment: Code coverage tool v1.2

Make work again after renaming the weave package to services-sync (bug 570636)

Comment 5

[Philipp von Weitershausen [:philikon]]

Attachment: Code coverage tool v1.3

Updated patch after bug 573668 moved the test setup around a bit. The coverage tool also lives in the 'tools' dir now.

Comment 6

[Philipp von Weitershausen [:philikon]]

Attachment: Code coverage tool v1.4

Latest Firefox nightlies report JS module filenames with the actual filenames but the resource:// URIs that they were imported under. To cope with that, the coverage filter can now be a whitespace-separated list of patterns so that we can filter for both resource://service-sync/* and the actual filename. Before writing the statistics to file we normalize everything to filenames. This also has the nice side effect that we now record coverage for XPCOM modules.

Comment 7

[Philipp von Weitershausen [:philikon]]

So it turns out the pure 'js' shell has a -D output option that prints out SpiderMonkey opcodes. David Flanagan has built a tool to analyze that output: https://github.com/davidflanagan/CoverMonkey. Would be cool to build on top of that, once we make 'xpcshell' support the -D option too :)

Comment 8

[Philipp von Weitershausen [:philikon]]

https://github.com/andreasgal/dom.js/issues/8 has a patch for making the JS shell write the opcode output to a file. This seems a useful thing to do when bringing coverage reporting to xpcshell.

Comment 9

[Gregory Szorc [:gps]]

It isn't stated explicitly in the bug (and the bug component disagrees), but is the intended scope of this bug for all xpcshell test suites or just Sync's? I would desire it to be for all, if possible.

Also, something to keep in mind is that there is a desire to execute xpcshell tests in parallel. This needs to be kept in mind when determining how the opcode output is handled (e.g. can't write to the same file from parallel processes without locking).

Comment 10

[Philipp von Weitershausen [:philikon]]

(In reply to Gregory Szorc [:gps] from comment #9)
> It isn't stated explicitly in the bug (and the bug component disagrees), but
> is the intended scope of this bug for all xpcshell test suites or just
> Sync's? I would desire it to be for all, if possible.

Yes, all xpcshell tests would be good. Easy enough to morph this bug.

> Also, something to keep in mind is that there is a desire to execute
> xpcshell tests in parallel. This needs to be kept in mind when determining
> how the opcode output is handled (e.g. can't write to the same file from
> parallel processes without locking).

This very much not a goal as there's no guarantee that tests don't already assume exclusive access to files, ports, etc.

Comment 11

[Gregory Szorc [:gps]]

(In reply to Philipp von Weitershausen [:philikon] from comment #10) 
> This very much not a goal as there's no guarantee that tests don't already
> assume exclusive access to files, ports, etc.

While this guarantee is true for Sync tests, it doesn't hold for all xpcshell test suites (like the JS JIT tests). I'm just saying that some parties may want code coverage without sacrificing parallel execution. That being said, I think it is fair for this to not be supported initially. But, if we can support it up front with minimal effort, we should strive for that.

Comment 12

[Mike Connor [:mconnor]]

Let's figure out scope here.  Do we want this for ourselves, or do we want to try to jump straight to a general-purpose solution.  If the latter, we should move this to Testing:Code Coverage

Comment 13

[Gregory Szorc [:gps]]

Not in philikon's court. I'd love to have a go at this. Can we get this on the roadmap?

Comment 14

[Lina Butler (ex-Mozilla)]

Firefox is getting tree-wide code coverage in bug 1278393.