1561004 - [meta] CSS coverage

Status: NEW

(Testing :: Code Coverage, task)

Description

[Marco Castelluccio [:marco]]

Meta bug to track the work required to collect coverage information for CSS files.

Comment 1

[Brian Grinstead [:bgrins]]

Even before we fix Bug 1561001 (stopping use of the preprocessor), there are components like devtools which don't typically use the preprocessor with CSS and could benefit from this. Some notes from an email thread about this project:

bgrins>

I’d like to get a sanity check on how feasible this feels. It seems like there are at least a few hard parts to start:

1. Getting information out of the style engine. Emilio/Cam, do we have a mechanism to track this in the parent process during a test run? Like, is it feasible to get a list of CSS rules (I guess mapped back to line numbers in the file) that were/weren’t used from a set of CSS files (like xul.css, global.css, browser.css) without major changes to the engine?
2. Mapping these rules back to the authored sources since we have preprocessing on some stylesheets. This might be more motivation to stop doing that and use imports on inline the rules.
3. Outputting this information into the correct code coverage format. Marco, is there any documentation or reference files to see here? Also, what tests are run during a ccov job? Are they running in multiple processes (chunked per folder or similar), or does everything happen in a single process?

marco>

grcov, which we use to parse the coverage files, currently supports INFO
files (which are textual format, so easier to generate for you) and
GCNO/GCDA files (binary format which the C/C++ compilers generate).

The easiest for you would probably be an INFO file (which is the same
the JS engine is producing), here's an example:
https://github.com/mozilla/grcov/blob/master/test/prova_fn_with_commas.info.
Documentation about the format is available at the end of the page at
http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php.

In coverage jobs, we run almost all tests we run with debug builds,
currently on Linux and Windows:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov.
The parallelization is also the same as normal debug builds, so
depending on the test suite they could be running in multiple processes
or in a single process.

A note: this work could also be interesting for integration into
DevTools, it could be surfaced in the style viewer. I can see web
developers using this kind of info.

emilio>

1. Getting information out of the style engine.

It's not terribly impossible. The "mapped back to line numbers" bits may
be hard, depending on whether you want lines for each declaration
separately or not. Would you want coverage for selectors matching
(easier)? Or for declarations applying (harder)? For example, if you have:

div { color: red; }
div { color: white; }

You'd get coverage for both lines if there's a <div> on the page for the
first model, while only for the second one if you track it at the
declaration level.

Other than that, the only question I have is whether a build-time switch
would be acceptable. Otherwise (2) may be a quite high price to pay
(since you need to add a byte to every declaration).

2. Mapping these rules back to the authored sources since we have preprocessing on some stylesheets. This might be more motivation to stop doing that and use imports on inline the rules.

This should be doable with source maps... We have source mapping
information, but IIRC devtools are the ones which actually pull the map
and do the conversion. Would it be feasible to make it a two-step
process, where the style engine gives you the "raw" positions, and
something parses it and generates the final source-mapped thing?

I also don't know whether generating sourcemaps that know about the
preprocessor is easy or not. But that looks like the best way forward to
me. May be useful to get feedback from devtools people.

3. Outputting this information into the correct code coverage format. Marco, is there any documentation or reference files to see here? Also, what tests are run during a ccov job? Are they running in multiple processes (chunked per folder or similar), or does everything happen in a single process?

Assuming we have the information I suspect this should be easy-ish to do :)